SambaStack Multi-tenancy with vCluster: Private Nodes

vCluster, vCluster Platform, and ingress-nginx are third-party products. SambaNova doesn’t maintain, endorse, or audit them. This reference architecture is provided for informational purposes and may become outdated as they change.

In the private-node model, a tenant’s worker nodes join the tenant’s control plane directly. They never become members of the host cluster, and no host component runs on them. The tenant control plane runs as a pod on a CPU node pool of the host, and the tenant brings its own kubelet, CNI, DNS, and scheduler, all installed by vCluster. Use this model for external, resold, or regulated tenants, and for any tenant that gets cluster-admin access. For trusted internal tenants, shared nodes need less setup. See the overview for the comparison. This guide joins RHEL 8.10 nodes to a tenant running Kubernetes 1.35. Each tenant gets two kinds of private node:

CPU nodes (x86_64, no RDUs) run the SambaStack platform services (API gateway, UI, authentication, Postgres, Redis), the ingress controller, and tenant DNS.
RDU nodes run only the model bundle pods.

Because the platform never runs on an RDU node, you can drain, reboot, or upgrade an RDU node without taking down the tenant’s API, UI, or login. Only the model replicas on that node move or stop.

Tested with: SambaStack 2.3.2, RKE2 v1.35.5 host, vCluster CLI and vcluster-pro 0.36.4, vCluster Platform 4.12.4 (free plan), tenant Kubernetes v1.35.10, RHEL 8.10 private nodes, containerd v2.1.6 (static build), ingress-nginx chart 4.15.1.The vCluster Platform free plan is capped at 64 CPUs or 32 GPUs across the instance. Larger deployments need a paid plan. See vCluster pricing.

vCluster’s standard node bundle for Kubernetes 1.35 doesn’t run on RHEL 8.10. Step 4 builds an RHEL 8 bundle that uses containerd’s static build instead. Validate the bundle on a test node before production.

Architecture

How the pieces work together:

Join. vcluster token create issues a bootstrap token. Each node runs the join script with the RHEL 8 bundle, which installs kubelet and containerd and joins the node to the tenant API server through the NodePort.
Control traffic. Every connection starts on the node and goes out to the tenant API server. The control plane never dials the node.
Logs and exec. A konnectivity agent on the node keeps a tunnel open to the konnectivity server in the control plane pod. The API server reaches the kubelet back through that tunnel.
Pod network. Flannel runs on the tenant’s own pod and service CIDRs. These never touch the host’s network.
App traffic. DNS sends the tenant’s SambaStack hostnames to the CPU nodes, where ingress-nginx in the host network terminates TLS and routes to SambaStack. SambaStack sends inference requests on to the model pods on the RDU nodes.
Placement. Node labels keep the two kinds apart. SambaStack’s own services, Postgres, and Redis select sambanova.ai/role=control-plane, so they run only on CPU nodes. SambaStack’s inference operator places model bundle pods by snRduArch, so they run only on RDU nodes.
License. The control plane pod gets its private-node entitlement from vCluster Platform on the host cluster.

Prerequisites

Host cluster, set up once:

A host cluster that meets the SambaStack on-prem requirements.
A CPU node pool labeled control-plane=shared for tenant control planes.
A storage class for tenant control plane state. This reference uses openebs-hostpath.
An ingress controller and a hostname for vCluster Platform, and egress from the host cluster to admin.loft.sh.
A way to expose each tenant API: a LoadBalancer service, or a NodePort on the host nodes as in this reference.

Per tenant, two kinds of private node:

CPU nodes: x86_64 (Intel or AMD) servers without RDUs, for the SambaStack platform. Use at least two so the platform survives the loss of one, and size them as in the SambaStack on-prem requirements.
RDU nodes: the tenant’s RDU servers, for model bundles only.

Every private node, CPU or RDU:

Runs RHEL 8.10 and is not a member of the host cluster. A node joins one cluster only.
cgroup v2 enabled. Kubernetes 1.35 kubelet and kubeadm refuse to run on cgroup v1, which is the RHEL 8 default. stat -fc %T /sys/fs/cgroup/ must print cgroup2fs.
No other kubelet, Docker, or containerd running.
Outbound TCP to a host node IP on the tenant’s NodePort (API server and konnectivity).
Access to github.com to download the Kubernetes bundle and containerd, or both files staged on the node.
If SELinux is Enforcing: access to rpm.vcluster.com, from which the join script installs the vcluster-selinux policy package.

Per tenant:

Unique hostnames and TLS certificates for the tenant’s SambaStack API and UI, as in the on-prem requirements. DNS for those hostnames points at the tenant’s CPU nodes, or at a VIP in front of them.
SambaStack 2.1.1 or later, which adds the global.nodeSelector setting used to pin the platform to CPU nodes.

On the operator’s machine: kubectl, helm v3.19+, and the vCluster CLI.

Components

Layer	Component	Provided by	Runs on
Container runtime, kubelet	kubelet v1.35.10, runc, CNI plugins; containerd v2.1.6 static build	RHEL 8 bundle (step 4), installed by the vCluster join script	Private node
SELinux policy	`vcluster-selinux` (el8)	vCluster join script, when SELinux is enabled	Private node
CNI, service proxy, metrics	Flannel, kube-proxy, metrics-server	vCluster (`deploy.*` settings)	Private node
DNS	CoreDNS	vCluster (`controlPlane.coredns`)	CPU nodes
Node tunnel	konnectivity agent	vCluster (`controlPlane.advanced.konnectivity`)	Every private node
Default storage class	local-path-provisioner	vCluster	Private node
Node role labels	`sambanova.ai/role=control-plane` on CPU nodes, `snRduArch=sn40-16` on RDU nodes	You, per node	Private node
Tenant control plane	API server, scheduler, controller manager, embedded etcd, konnectivity server	vCluster Platform, `vcluster create`	Host namespace `vcluster-tenant1`, CPU node pool
Private node licensing	vCluster Platform, free plan	You, once per host	Host
Tenant app ingress	ingress-nginx	You, per tenant	CPU nodes, host network
SambaStack platform	`sambastack-base`, `sambastack`: gateway, UI, auth, Postgres, Redis, inference operator	You, per tenant	CPU nodes
Model bundles	`sambastack-models`	You, per tenant	RDU nodes

Set up a tenant

Commands run from the operator’s machine unless noted. Replace tenant1, vcluster-tenant1, <host-node-ip>, <platform-hostname>, and <node-name> with your values.

Install vCluster Platform (once per host)

Private nodes aren’t in open-source vCluster. They need vCluster Platform. The free plan includes private nodes but is capped at 64 CPUs or 32 GPUs across the instance; check the vCluster pricing page against your node sizes.The install must set config.loftHost to a hostname that resolves from your machine, from the host nodes, and from pods.

platform-values.yaml

config:
  loftHost: <platform-hostname>
ingress:
  enabled: true
  host: <platform-hostname>
  ingressClass: nginx
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"

vcluster platform start --namespace vcluster-platform --email <admin-email> \
  --no-tunnel --values platform-values.yaml --version 4.12.4

Then log in to the Platform UI at https://<platform-hostname> as admin and activate the free plan. vcluster platform start prints the admin password; to read it later, run helm get values loft -n vcluster-platform against the host. Until the instance is activated, vcluster create fails. Log in from the CLI with the external URL, not a local port-forward, so that tenant control planes can reach vCluster Platform:

vcluster platform login https://<platform-hostname> --insecure

--insecure is for a self-signed Platform certificate. Drop it if the hostname has a trusted certificate.

Create the tenant control plane

The settings that carry the design:

Setting	Value	Why
`privateNodes.enabled`	`true`	Worker nodes join the tenant directly. Can’t be changed after creation
`controlPlane.distro.k8s.image.tag`	`v1.35.10`	Pins the tenant to Kubernetes 1.35. The join script downloads the node bundle for this version, and the bundle you build in step 4 must match it
`controlPlane.service`	`NodePort` 31443	Exposes the tenant API on every host node IP. Use `LoadBalancer` if your host has one
`controlPlane.endpoint`	`<host-node-ip>:31443`	The address the join script and kubelets use
`controlPlane.proxy.extraSANs`	every host node IP, or the VIP	The API certificate is valid for whichever address the nodes use
`controlPlane.statefulSet.scheduling.nodeSelector`	`control-plane: shared`	The control plane runs on the host’s CPU pool, not on RDU nodes
`controlPlane.coredns.deployment.nodeSelector`	`sambanova.ai/role: control-plane`	Tenant DNS runs on the tenant’s CPU nodes, so RDU maintenance doesn’t interrupt name resolution
`networking.podCIDR`, `serviceCIDR`	tenant-only ranges	Must not overlap with the host or other tenants
`deploy.cni.flannel`, `kubeProxy`, `localPathProvisioner`, `metricsServer`	`enabled: true`	vCluster installs the node-side basics

vcluster.yaml

privateNodes:
  enabled: true
controlPlane:
  distro:
    k8s:
      enabled: true
      image:
        tag: v1.35.10
  coredns:
    enabled: true
    deployment:
      replicas: 2
      nodeSelector:
        sambanova.ai/role: control-plane
  backingStore:
    etcd:
      embedded:
        enabled: true
  statefulSet:
    persistence:
      volumeClaim:
        enabled: true
        storageClass: openebs-hostpath
    scheduling:
      nodeSelector:
        control-plane: shared
  service:
    spec:
      type: NodePort
      ports:
        - name: https
          port: 443
          targetPort: 8443
          nodePort: 31443
          protocol: TCP
  endpoint: <host-node-ip>:31443
  proxy:
    extraSANs:
      - <host-node-ip-1>
      - <host-node-ip-2>
      - <host-node-ip-3>
networking:
  podCIDR: 10.64.0.0/16
  serviceCIDR: 10.128.0.0/16
deploy:
  cni:
    flannel:
      enabled: true
  kubeProxy:
    enabled: true
  localPathProvisioner:
    enabled: true
  metricsServer:
    enabled: true

If the host’s CPU node pool is tainted, add matching controlPlane.statefulSet.scheduling.tolerations, or the control plane pod stays Pending. CoreDNS stays Pending until the first tenant CPU node is joined and labeled in step 6; that is expected.

vcluster create tenant1 -n vcluster-tenant1 -f vcluster.yaml --connect=false --driver helm

vcluster connect tenant1 -n vcluster-tenant1 --print \
  --server https://<host-node-ip>:31443 > tenant1.kubeconfig

The kubeconfig holds a cluster-admin client certificate for the tenant. Treat it like a password.

Prepare the RHEL 8.10 nodes

Do this on every CPU node and every RDU node, as root. First confirm cgroup v2; stop here if it prints tmpfs:

stat -fc %T /sys/fs/cgroup/        # must print cgroup2fs

Stop any existing container runtime, turn off swap, and load the kernel modules and sysctls. The files under /etc keep them across reboots:

systemctl disable --now docker.socket docker containerd 2>/dev/null || true

swapoff -a
sed -i '/\sswap\s/s/^/#/' /etc/fstab

cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
bridge
br_netfilter
vxlan
nf_conntrack
EOF
modprobe -a overlay bridge br_netfilter vxlan nf_conntrack

cat <<EOF > /etc/sysctl.d/99-k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

dnf install -y iptables curl tar conntrack-tools socat

Build the RHEL 8 node bundle

Build the bundle once, on any one of the RHEL 8.10 nodes, as root. It replaces the bundle’s containerd binaries with containerd’s static build of the same version:

cd /root
curl -fsSLO https://github.com/loft-sh/kubernetes/releases/download/v1.35.10/kubernetes-v1.35.10-amd64.tar.gz
curl -fsSLO https://github.com/containerd/containerd/releases/download/v2.1.6/containerd-static-2.1.6-linux-amd64.tar.gz
echo "577900a5a8684c27e344aeeb1fc64e355745f58cba7f83c53649235ba25abbbf  containerd-static-2.1.6-linux-amd64.tar.gz" | sha256sum -c -

mkdir -p bundle containerd-static
tar -xzf kubernetes-v1.35.10-amd64.tar.gz -C bundle
tar -xzf containerd-static-2.1.6-linux-amd64.tar.gz -C containerd-static
install -m 0755 containerd-static/bin/containerd containerd-static/bin/containerd-shim-runc-v2 \
   containerd-static/bin/ctr bundle/release/

file bundle/release/containerd bundle/release/ctr bundle/release/containerd-shim-runc-v2   # each: statically linked
bundle/release/containerd --version   # containerd ... v2.1.6

tar -czf kubernetes-v1.35.10-amd64-el8.tar.gz -C bundle release

Copy /root/kubernetes-v1.35.10-amd64-el8.tar.gz to /root on every other CPU and RDU node. The bundle version must match the tenant’s Kubernetes version (controlPlane.distro.k8s.image.tag); build a new bundle when you upgrade the tenant.

Join the nodes

Join the CPU nodes first, then the RDU nodes. Each node joins the same way. From your machine, with the tenant kubeconfig, issue a join token:

KUBECONFIG=tenant1.kubeconfig vcluster token create --expires=4h

The command prints a curl ... | sh - join command. vCluster prints it with curl -k, which skips verification of the tenant API server’s certificate. Run it on each node as root, adding --node-name and --bundle-path:

curl -fsSLk "https://<host-node-ip>:31443/node/join?token=<token>" | sh -s -- \
  --node-name <node-name> --bundle-path /root/kubernetes-v1.35.10-amd64-el8.tar.gz

The join ends with This node has joined the cluster. One token joins several nodes until it expires.With SELinux Enforcing, the join installs the vcluster-selinux package from rpm.vcluster.com. On nodes without access to it, install the package first and add --skip-selinux-rpm to the join command.

Label the nodes

The labels decide what runs where. CPU nodes get the platform label, and RDU nodes get the RDU label that SambaStack’s inference operator selects:

export KUBECONFIG=tenant1.kubeconfig

# CPU nodes: SambaStack platform, ingress, DNS
kubectl label nodes <cpu-node-1> <cpu-node-2> sambanova.ai/role=control-plane --overwrite

# RDU nodes: model bundles only
kubectl label nodes <rdu-node-1> <rdu-node-2> snRduArch=sn40-16 --overwrite

kubectl get nodes -L sambanova.ai/role,snRduArch

Never put both labels on one node. CoreDNS, which was Pending, starts on the CPU nodes once they are labeled.

Install an ingress controller in the tenant

Private nodes have no host ingress controller, so the tenant needs its own. This install runs ingress-nginx only on the CPU nodes and binds it to ports 80 and 443 of their host network, so DNS for the tenant’s SambaStack hostnames can point at the CPU node IPs.

export KUBECONFIG=tenant1.kubeconfig
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx --version 4.15.1 \
  -n ingress-nginx --create-namespace \
  --set controller.kind=DaemonSet --set controller.hostNetwork=true \
  --set controller.dnsPolicy=ClusterFirstWithHostNet --set controller.service.type=ClusterIP \
  --set controller.ingressClassResource.default=true --set controller.admissionWebhooks.enabled=false \
  --set-string 'controller.nodeSelector.sambanova\.ai/role=control-plane'

Any ingress controller that serves the node’s ports 80 and 443 and handles standard Ingress objects works in place of ingress-nginx.

Verify

From the tenant:

export KUBECONFIG=tenant1.kubeconfig
kubectl get nodes -o wide -L sambanova.ai/role,snRduArch   # CPU and RDU nodes, Ready, v1.35.10, RHEL 8.10
kubectl -n kube-system get pods -o wide       # flannel, kube-proxy, konnectivity agent on every node; CoreDNS on CPU nodes
kubectl top node                              # metrics-server works through the tunnel

On the node:

containerd --version                          # v2.1.6
systemctl is-active containerd kubelet        # active, active

From the host:

unset KUBECONFIG
kubectl get nodes                              # the private node is not listed
kubectl -n vcluster-tenant1 get pods           # only the control plane pod, tenant1-0

To check the tunnel, run a test deployment in the tenant and confirm that kubectl logs and kubectl exec work against it.

Install SambaStack in the tenant

Install SambaStack with the standard on-prem quickstart, using the tenant kubeconfig (export KUBECONFIG=tenant1.kubeconfig). SambaNova provides the registry URL, the service-account key and the chart version. Only these settings differ from a normal install. Node selectors. The node labels from step 6 decide where each pod runs:

global.nodeSelector sends every SambaStack service (gateway, UI, authentication, inference operator, and the rest) to the CPU nodes labeled sambanova.ai/role=control-plane.
Postgres, Redis, and OpenEBS come from bundled charts that don’t read the global setting, so each gets the same selector in its own section.
Model bundle pods aren’t affected. SambaStack’s inference operator places them on the RDU nodes labeled snRduArch=sn40-16.

Add these to sambastack.yaml. Requires SambaStack 2.1.1 or later. For the per-service overrides, tolerations, and the other scheduling settings, see the SambaStack Helm values file and the SambaStack v2.1.1 release notes.

sambastack.yaml (tenant additions)

global:
  nodeSelector:
    sambanova.ai/role: control-plane
cloudnative-pg:
  nodeSelector:
    sambanova.ai/role: control-plane
  clusterSpec:
    affinity:
      nodeSelector:
        sambanova.ai/role: control-plane
global-queue-redis:
  master:
    nodeSelector:
      sambanova.ai/role: control-plane
response-queue-redis:
  master:
    nodeSelector:
      sambanova.ai/role: control-plane
openebs:
  localpv-provisioner:
    localpv:
      nodeSelector:
        sambanova.ai/role: control-plane

Skip the quickstart’s node labeling step; the nodes were labeled in step 6. Storage classes. SambaStack installs its own OpenEBS provisioner and stores Postgres data in the openebs-hostpath class, under /scratch/openebs. With the selectors above, that path is on the CPU nodes, so give them enough disk there, or set openebs.localpv-provisioner.hostpathClass.basePath. Ingress and DNS. Use the ingress-nginx controller from step 7, which is the tenant’s default ingress class. Point the DNS records for the SambaStack API and UI hostnames at the CPU node IPs, or a VIP in front of them, never at RDU nodes. The hostnames must also resolve from inside the tenant, and the TLS certificate must come from a trusted CA, as the quickstart requires. After the install, check that the platform pods are on CPU nodes and the model pods on RDU nodes:

kubectl -n sambastack get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName | sort -k2
kubectl get nodes -L sambanova.ai/role,snRduArch

From the host (with KUBECONFIG unset), nothing changes: kubectl -n vcluster-tenant1 get pods still shows only the control plane pod.

Maintain an RDU node

Because the platform runs on CPU nodes, RDU maintenance only affects the model replicas on that node. Plan at least two RDU nodes per model if it must keep serving during maintenance.

export KUBECONFIG=tenant1.kubeconfig

# Stop new pods landing on the node and move the model pods off it
kubectl cordon <rdu-node>
kubectl drain <rdu-node> --ignore-daemonsets --delete-emptydir-data

# ... firmware, OS, or RDU runtime work, reboot ...

kubectl uncordon <rdu-node>

The gateway, UI, login, Postgres, Redis, and DNS keep running throughout. Drain CPU nodes one at a time, and only while the others are Ready.

Limitations

This reference is tested with RHEL 8.10 private nodes only. It isn’t tested with RHEL 10 nodes.
RHEL 8.10 nodes need the RHEL 8 bundle from step 4. Rebuild it for every tenant Kubernetes upgrade, and validate it on a test node first.
vCluster’s node-to-node VPN isn’t available on RHEL 8.10 nodes. This reference doesn’t use it.
The join token expires. Issue a new one for each additional node.
The tenant API sits on one NodePort of the host nodes. Use a LoadBalancer service or a VIP for production.
vCluster Platform must reach admin.loft.sh to validate the free plan, and the free plan’s CPU and GPU caps apply.
Tenant control plane state is on one volume. For production, back it up or run vCluster’s high-availability configuration.

Cleanup

# Tenant: remove SambaStack and the ingress controller
export KUBECONFIG=tenant1.kubeconfig
helm uninstall sambastack-models sambastack sambastack-base -n sambastack
helm uninstall ingress-nginx -n ingress-nginx

# Node, as root: remove the node components (needs a fresh token)
curl -fsSLk "https://<host-node-ip>:31443/node/join?token=<token>" | sh -s -- --reset-only
rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/99-k8s.conf

# Host: remove the tenant
unset KUBECONFIG
vcluster delete tenant1 -n vcluster-tenant1

Platform stays for other tenants. To remove it as well: helm uninstall loft -n vcluster-platform.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Private nodes

Architecture

Prerequisites

Components

Set up a tenant

Install vCluster Platform (once per host)

Create the tenant control plane

Prepare the RHEL 8.10 nodes

Build the RHEL 8 node bundle

Join the nodes

Label the nodes

Install an ingress controller in the tenant

Verify

Install SambaStack in the tenant

Maintain an RDU node

Limitations

Cleanup

Additional resources

​Architecture

​Prerequisites

​Components

​Set up a tenant

Install vCluster Platform (once per host)

Create the tenant control plane

Prepare the RHEL 8.10 nodes

Build the RHEL 8 node bundle

Join the nodes

Label the nodes

Install an ingress controller in the tenant

​Verify

​Install SambaStack in the tenant

​Maintain an RDU node

​Limitations

​Cleanup

​Additional resources

Architecture

Prerequisites

Components

Set up a tenant

Verify

Install SambaStack in the tenant

Maintain an RDU node

Limitations

Cleanup

Additional resources