SambaStack Multi-tenancy with vCluster: Shared Nodes

vCluster and Envoy Gateway are third-party products. SambaNova doesn’t maintain, endorse, or audit them. This reference architecture is provided for informational purposes and may become outdated as they change.

In the shared-node model, a tenant’s RDU nodes stay members of the host cluster. You label each node for one tenant, and vCluster shows the tenant only those nodes and places its pods only there. The tenant control plane runs as pods on a CPU node pool of the host. Nothing is installed on the RDU nodes. Use this model for trusted tenants inside your organization. For external or untrusted tenants, use private nodes. See the overview for the comparison.

Tested with: SambaStack 2.3.2, RKE2 v1.35.5, vCluster Helm chart 0.35.4 (open-source image), tenant Kubernetes v1.35.0, Envoy Gateway v1.9.1.

Architecture

How the pieces work together:

Claim. You label a host RDU node vcluster=tenant1. The tenant’s node sync selector matches that label, so the tenant sees only its nodes, and the syncer sets the same node selector on every pod it copies to the host.
Control traffic. kubectl connects to the tenant hostname on the gateway’s NodePort. The gateway reads the SNI hostname and passes the connection, still encrypted, to the tenant’s API server service. The tenant’s API certificate includes the hostname.
Pods. The tenant creates pods in its own API server. The syncer copies them into the host namespace vcluster-tenant1 under rewritten names, and the host scheduler places them, since the tenant’s own scheduler is off.
Logs and exec. The vCluster syncer forwards these to the host API server for the synced pod. No node agent is needed.
Pod network. Tenant pods use the host CNI and the host’s pod and service CIDRs.
App traffic. Tenant Ingress objects sync to the host, and the host’s ingress controller serves them.

Prerequisites

Host cluster, set up once:

A host cluster that meets the SambaStack on-prem requirements, with the RDU nodes already labeled snRduArch=sn40-16 as that guide describes.
A CPU node pool labeled control-plane=shared for tenant control planes. Without it the tenant’s API server and etcd pods stay Pending.
A storage class for tenant control plane state. This reference uses openebs-hostpath.
A gateway or load balancer that routes TLS by SNI hostname without terminating it. This reference uses Envoy Gateway. NGINX or HAProxy in passthrough mode also work.
A DNS name per tenant that resolves to the gateway, for example tenant1-api.example.com.

Per tenant:

One or more RDU worker nodes that are Ready and not labeled for another tenant.
Unique hostnames and TLS certificates for the tenant’s SambaStack API and UI, as in the on-prem requirements.

On the operator’s machine: kubectl, helm v3.19+, and the vCluster CLI.

Components

Layer	Component	Provided by	Runs on
Kubelet, container runtime, CNI, kube-proxy	RKE2	Host cluster	RDU node
Scheduler for tenant pods	kube-scheduler	Host cluster	Host control plane
RDU node software and labels	SambaStack host install	Host cluster	RDU node
Tenant control plane	API server, controller manager, syncer, etcd	vCluster chart	Host namespace `vcluster-tenant1`, CPU node pool
Tenant DNS	CoreDNS	vCluster chart	Synced pod on the tenant’s RDU node
Tenant API entry point	Envoy Gateway, `Gateway`, `TLSRoute`	You, once per host	Host
Tenant app ingress	Host ingress controller, through Ingress sync	Host cluster	Host
SambaStack	`sambastack-base`, `sambastack`, `sambastack-models`	You, per tenant	Tenant, synced to the RDU node

Set up a tenant

Commands run against the host cluster unless noted. Replace tenant1, vcluster-tenant1, tenant1-api.example.com, and <rdu-node> with your tenant’s values.

Envoy Gateway is the gateway this reference was tested with, not a requirement. Any gateway or load balancer that routes TLS by SNI hostname without terminating it works in its place. Steps 1 and 2 are specific to Envoy Gateway, and controlPlane.tlsRoute in step 4 creates a Gateway API TLSRoute. With a gateway that doesn’t implement the Gateway API, leave tlsRoute out and route the tenant hostname to the tenant1 service on port 443 in that gateway’s own configuration.

Install the gateway (once per host)

helm upgrade --install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  --version v1.9.1 \
  --namespace envoy-gateway-system --create-namespace
kubectl wait -n envoy-gateway-system deploy/envoy-gateway --for=condition=Available --timeout=5m

Create the TLS passthrough listener (once per host)

One listener on port 8443 in passthrough mode, exposed as NodePort 30843. Every tenant API shares it, routed by hostname.

gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: envoy-gateway-proxy-config
  listeners:
    - name: vcluster-api
      protocol: TLS
      port: 8443
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-gateway-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        externalTrafficPolicy: Cluster
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
                - name: vcluster-api
                  port: 8443
                  targetPort: 8443
                  nodePort: 30843
                  protocol: TCP

kubectl apply -f gateway.yaml

Label the tenant's RDU nodes

# Confirm the node is Ready, a worker, and not labeled for another tenant
kubectl get node <rdu-node> --show-labels | tr ',' '\n' | grep -E 'node-role|vcluster|snRduArch'

# Claim it for the tenant
kubectl label node <rdu-node> vcluster=tenant1 --overwrite

# Verify the tenant's nodes and the control plane pool
kubectl get nodes -l vcluster=tenant1
kubectl get nodes -l control-plane=shared

Write the tenant's vCluster values

The settings that carry the design:

Setting	Value	Why
`controlPlane.proxy.extraSANs`	`tenant1-api.example.com`	The API certificate is valid for the tenant hostname
`controlPlane.tlsRoute`	host `tenant1-api.example.com`, parent `envoy-gateway`	The gateway routes this hostname to this tenant
`controlPlane.statefulSet.scheduling.nodeSelector`	`control-plane: shared`	API server and etcd run on the CPU pool, not on RDU nodes
`controlPlane.distro.k8s.scheduler.enabled`	`false`	The host scheduler places synced pods
`controlPlane.coredns.deployment.nodeSelector`	`vcluster: tenant1`	Tenant DNS runs on the tenant’s node
`sync.fromHost.nodes.selector.labels`	`vcluster: tenant1`	The tenant sees only the nodes with this label, and synced pods get the same node selector
`sync.fromHost.nodes.syncBackChanges`	`false`	Tenants can’t edit host node labels or taints, including their own `vcluster` label. See the warning below
`sync.toHost.ingresses`	`true`	Tenant Ingresses are served by the host ingress controller

vcluster.yaml

controlPlane:
  proxy:
    extraSANs:
      - tenant1-api.example.com
      - tenant1.vcluster-tenant1.svc.cluster.local
  tlsRoute:
    enabled: true
    host: tenant1-api.example.com
    parentRefs:
      - name: envoy-gateway
        namespace: envoy-gateway-system
  statefulSet:
    highAvailability:
      replicas: 1
    image:
      registry: ghcr.io
      repository: loft-sh/vcluster-oss
    persistence:
      volumeClaim:
        enabled: true
        retentionPolicy: Delete
        storageClass: openebs-hostpath
    scheduling:
      nodeSelector:
        control-plane: shared
      podManagementPolicy: Parallel
      tolerations:
        - operator: Exists
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 1
          persistence:
            volumeClaim:
              accessModes: [ReadWriteOnce]
              enabled: true
              retentionPolicy: Delete
              size: 10Gi
              storageClass: openebs-hostpath
          scheduling:
            nodeSelector:
              control-plane: shared
            podManagementPolicy: Parallel
            tolerations:
              - operator: Exists
  coredns:
    enabled: true
    deployment:
      nodeSelector:
        vcluster: tenant1
      replicas: 1
  distro:
    k8s:
      enabled: true
      image:
        registry: ghcr.io
        repository: loft-sh/kubernetes
        tag: v1.35.0
      apiServer:
        enabled: true
        extraArgs:
          - --enable-admission-plugins=PodNodeSelector,PodTolerationRestriction
      controllerManager:
        enabled: true
      scheduler:
        enabled: false
networking:
  advanced:
    fallbackHostCluster: true
sync:
  fromHost:
    csiDrivers: {enabled: auto}
    csiNodes: {enabled: auto}
    csiStorageCapacities: {enabled: auto}
    events: {enabled: true}
    ingressClasses: {enabled: false}
    storageClasses: {enabled: auto}
    nodes:
      enabled: true
      clearImageStatus: false
      syncBackChanges: false
      selector:
        all: false
        labels:
          vcluster: tenant1
  toHost:
    configMaps: {all: false, enabled: true}
    endpoints: {enabled: true}
    ingresses: {enabled: true}
    networkPolicies: {enabled: false}
    persistentVolumeClaims: {enabled: true}
    persistentVolumes: {enabled: false}
    podDisruptionBudgets: {enabled: false}
    priorityClasses: {enabled: false}
    secrets: {all: false, enabled: true}
    serviceAccounts: {enabled: false}
    services: {enabled: true}
    storageClasses: {enabled: true}
    pods:
      enabled: true
      rewriteHosts:
        enabled: true

PodNodeSelector and PodTolerationRestriction let you pin a tenant namespace to specific nodes or tolerations through namespace annotations. They are optional; the node sync selector already confines tenant pods to labeled nodes.

Tooling that labels nodes from inside the tenant, for example to drive a rolling runtime upgrade node by node, needs syncBackChanges: true. That setting also lets a tenant administrator change or remove any label or taint on its host nodes, including the vcluster label that confines it. Enable it only for tenants you trust to manage their own nodes, and keep the tenant’s cluster-admin kubeconfig with the platform team.

Install the tenant

helm repo add loft-sh https://charts.loft.sh && helm repo update

helm upgrade --install tenant1 loft-sh/vcluster \
  --version 0.35.4 \
  --namespace vcluster-tenant1 --create-namespace \
  -f vcluster.yaml

kubectl -n vcluster-tenant1 get pods -w

tenant1-etcd-0 comes up first. tenant1-0, which runs the API server and syncer, is Running about a minute after the install.

Create DNS and connect

Create a DNS record for tenant1-api.example.com that resolves to a host node IP or to a VIP in front of the host nodes. Then write a tenant kubeconfig that points at the gateway’s NodePort:

vcluster connect tenant1 -n vcluster-tenant1 --print \
  --server https://tenant1-api.example.com:30843 > tenant1.kubeconfig

export KUBECONFIG=tenant1.kubeconfig
kubectl get nodes

kubectl get nodes returns only the nodes labeled vcluster=tenant1.

The kubeconfig holds a cluster-admin client certificate for the tenant. Treat it like a password. Anyone with access to the host cluster can generate one with the same command.

Verify

From the tenant:

export KUBECONFIG=tenant1.kubeconfig
kubectl get nodes -o wide          # only the tenant's labeled nodes
kubectl -n kube-system get pods     # CoreDNS, scheduled on the tenant's node

From the host:

unset KUBECONFIG
kubectl -n vcluster-tenant1 get pods -o wide

The host namespace shows the control plane pods on the CPU pool and the synced tenant pods, under rewritten names, on the labeled RDU node.

Install SambaStack in the tenant

Install SambaStack with the standard on-prem quickstart, using the tenant kubeconfig (export KUBECONFIG=tenant1.kubeconfig). Only these points differ from a normal install:

Skip the quickstart’s node labeling step. The snRduArch=sn40-16 label is set on the host and reaches the tenant through node sync.
Ingress and DNS. Tenant Ingress objects sync to the host and the host’s ingress controller serves them. Point the tenant’s API and UI hostnames at the host ingress, and keep them unique across tenants. The hostnames must also resolve from inside the tenant, and the TLS certificate must come from a trusted CA, as the quickstart requires.

From the host, the tenant’s pods appear in vcluster-tenant1 under rewritten names, all on the tenant’s labeled nodes: kubectl -n vcluster-tenant1 get pods -o wide.

Limitations

The vcluster=tenant1 label is a scheduling rule, not a security boundary. A tenant with cluster-admin access to its virtual cluster shares the host’s kernel, CNI, and kubelets. Use private nodes for those tenants.
The SambaStack platform services run on the tenant’s RDU nodes, so maintenance on those nodes interrupts the tenant’s API, UI, and login. To keep them apart, use private nodes with dedicated CPU nodes.
Host administrators see the tenant’s pods in the host namespace, and host DaemonSets such as monitoring agents run on the tenant’s nodes.
With syncBackChanges: true, a tenant administrator can edit its host nodes’ labels and taints. This reference sets it to false; see the warning in the values step.
Tenant pods share the host pod network. To restrict traffic between tenants, apply NetworkPolicy resources in each tenant’s host namespace, or enable sync.toHost.networkPolicies so tenants manage their own.
This reference runs one replica each of the API server and etcd on host-path storage. For production, raise highAvailability.replicas and back up the etcd volume.
The tenant API is reached through one NodePort on one host node IP. Put a VIP or load balancer in front of the host nodes for production.

Cleanup

# Tenant: remove SambaStack first
export KUBECONFIG=tenant1.kubeconfig
helm uninstall sambastack-models sambastack sambastack-base -n sambastack

# Host: remove the tenant and release the node
unset KUBECONFIG
helm uninstall tenant1 -n vcluster-tenant1
kubectl delete namespace vcluster-tenant1
kubectl label node <rdu-node> vcluster-

With retentionPolicy: Delete, the control plane volumes and all tenant state are removed with the Helm release. The gateway stays for other tenants.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Shared nodes

Architecture

Prerequisites

Components

Set up a tenant

Install the gateway (once per host)

Create the TLS passthrough listener (once per host)

Label the tenant's RDU nodes

Write the tenant's vCluster values

Install the tenant

Create DNS and connect

Verify

Install SambaStack in the tenant

Limitations

Cleanup

Additional resources

​Architecture

​Prerequisites

​Components

​Set up a tenant

Install the gateway (once per host)

Create the TLS passthrough listener (once per host)

Label the tenant's RDU nodes

Write the tenant's vCluster values

Install the tenant

Create DNS and connect

​Verify

​Install SambaStack in the tenant

​Limitations

​Cleanup

​Additional resources

Architecture

Prerequisites

Components

Set up a tenant

Verify

Install SambaStack in the tenant

Limitations

Cleanup

Additional resources