> ## Documentation Index > Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Private nodes > [!NOTE] > > vCluster, vCluster Platform, and ingress-nginx are third-party products. SambaNova doesn't maintain, endorse, or audit them. This reference architecture is provided for informational purposes and may become outdated as they change. In the private-node model, a tenant's worker nodes join the tenant's control plane directly. They never become members of the host cluster, and no host component runs on them. The tenant control plane runs as a pod on a CPU node pool of the host, and the tenant brings its own kubelet, CNI, DNS, and scheduler, all installed by vCluster. Use this model for external, resold, or regulated tenants, and for any tenant that gets cluster-admin access. For trusted internal tenants, [shared nodes](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/reference-architecture/multi-tenancy/shared-nodes.md) need less setup. See the [overview](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/reference-architecture/multi-tenancy/overview.md) for the comparison. This guide joins **RHEL 8.10** nodes to a tenant running **Kubernetes 1.35**. Each tenant gets two kinds of private node: * **CPU nodes** (x86\_64, no RDUs) run the SambaStack platform services (API gateway, UI, authentication, Postgres, Redis), the ingress controller, and tenant DNS. * **RDU nodes** run only the model bundle pods. Because the platform never runs on an RDU node, you can drain, reboot, or upgrade an RDU node without taking down the tenant's API, UI, or login. Only the model replicas on that node move or stop. > [!NOTE] > > **Tested with:** SambaStack 2.3.2, RKE2 v1.35.5 host, vCluster CLI and `vcluster-pro` 0.36.4, vCluster Platform 4.12.4 (free plan), tenant Kubernetes v1.35.10, RHEL 8.10 private nodes, containerd v2.1.6 (static build), ingress-nginx chart 4.15.1. > > The vCluster Platform free plan is capped at 64 CPUs or 32 GPUs across the instance. Larger deployments need a paid plan. See [vCluster pricing](https://www.vcluster.com/pricing). > [!NOTE] > > vCluster's standard node bundle for Kubernetes 1.35 doesn't run on RHEL 8.10. Step 4 builds an RHEL 8 bundle that uses containerd's static build instead. Validate the bundle on a test node before production. ## Architecture ```mermaid theme={} graph TB accTitle: Private-node tenant on a SambaStack host cluster accDescr: The tenant control plane runs as a pod in a host namespace and is exposed on a NodePort. Two kinds of RHEL 8.10 private node outside the host cluster join it, each running kubelet and containerd from the RHEL 8 bundle and opening all connections outward to the NodePort. CPU nodes run the ingress controller, CoreDNS, and the SambaStack platform services. RDU nodes run only model bundle pods. vCluster Platform on the host licenses private nodes. OP["Operator
kubectl · host-node-ip:31443"] subgraph HOST["Host cluster"] PLAT["vCluster Platform
free plan
licenses private nodes"] subgraph NS["Namespace vcluster-tenant1 · CPU node pool"] SVC["Service tenant1
NodePort 31443"] CP["Tenant control plane · K8s v1.35.10
API server · scheduler
controller manager · etcd
konnectivity server"] end end subgraph CPUN["CPU nodes · x86_64 · label sambanova.ai/role=control-plane"] ING["ingress-nginx
host network, port 443
api.tenant1.example.com
ui.tenant1.example.com"] SS["SambaStack platform
gateway · UI · auth
Postgres · Redis · inf-operator"] DNS["CoreDNS"] end subgraph RDUN["RDU nodes · label snRduArch=sn40-16"] MB["Model bundle pods"] end NOTE["Every private node: RHEL 8.10 · cgroup v2
kubelet v1.35.10 · containerd v2.1.6 static
Flannel · kube-proxy · konnectivity agent"] OP -->|"HTTPS"| SVC PLAT -.->|"license"| CP SVC --> CP CP <-->|"API and konnectivity tunnel,
opened outbound by each node"| CPUN CP <--> RDUN ING -->|"HTTPS from SambaStack users"| SS SS -->|"inference requests"| MB RDUN ~~~ NOTE classDef host fill:#eef2f7,stroke:#8aa0bd,color:#1a2b45; classDef tenant fill:#cfe2f3,stroke:#2f6fb0,color:#0d2c4d,stroke-width:2px; classDef rdu fill:#fde9cf,stroke:#c07a1c,color:#4a2c05,stroke-width:2px; classDef note fill:#ffffff,stroke:#8aa0bd,color:#1a2b45,stroke-dasharray:4 3; class PLAT,HOST host; class CP,SVC,NS,CPUN,ING,SS,DNS tenant; class RDUN,MB rdu; class NOTE note; ``` How the pieces work together: 1. **Join.** `vcluster token create` issues a bootstrap token. Each node runs the join script with the RHEL 8 bundle, which installs kubelet and containerd and joins the node to the tenant API server through the NodePort. 2. **Control traffic.** Every connection starts on the node and goes out to the tenant API server. The control plane never dials the node. 3. **Logs and exec.** A konnectivity agent on the node keeps a tunnel open to the konnectivity server in the control plane pod. The API server reaches the kubelet back through that tunnel. 4. **Pod network.** Flannel runs on the tenant's own pod and service CIDRs. These never touch the host's network. 5. **App traffic.** DNS sends the tenant's SambaStack hostnames to the CPU nodes, where ingress-nginx in the host network terminates TLS and routes to SambaStack. SambaStack sends inference requests on to the model pods on the RDU nodes. 6. **Placement.** Node labels keep the two kinds apart. SambaStack's own services, Postgres, and Redis select `sambanova.ai/role=control-plane`, so they run only on CPU nodes. SambaStack's inference operator places model bundle pods by `snRduArch`, so they run only on RDU nodes. 7. **License.** The control plane pod gets its private-node entitlement from vCluster Platform on the host cluster. ## Prerequisites Host cluster, set up once: * A host cluster that meets the SambaStack [on-prem requirements](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/getting-started/on-prem.md#prerequisites). * A CPU node pool labeled `control-plane=shared` for tenant control planes. * A storage class for tenant control plane state. This reference uses `openebs-hostpath`. * An ingress controller and a hostname for vCluster Platform, and egress from the host cluster to `admin.loft.sh`. * A way to expose each tenant API: a `LoadBalancer` service, or a NodePort on the host nodes as in this reference. Per tenant, two kinds of private node: * **CPU nodes:** x86\_64 (Intel or AMD) servers without RDUs, for the SambaStack platform. Use at least two so the platform survives the loss of one, and size them as in the SambaStack on-prem requirements. * **RDU nodes:** the tenant's RDU servers, for model bundles only. Every private node, CPU or RDU: * Runs **RHEL 8.10** and is not a member of the host cluster. A node joins one cluster only. * **cgroup v2 enabled.** Kubernetes 1.35 kubelet and kubeadm refuse to run on cgroup v1, which is the RHEL 8 default. `stat -fc %T /sys/fs/cgroup/` must print `cgroup2fs`. * No other kubelet, Docker, or containerd running. * Outbound TCP to a host node IP on the tenant's NodePort (API server and konnectivity). * Access to `github.com` to download the Kubernetes bundle and containerd, or both files staged on the node. * If SELinux is `Enforcing`: access to `rpm.vcluster.com`, from which the join script installs the `vcluster-selinux` policy package. Per tenant: * Unique hostnames and TLS certificates for the tenant's SambaStack API and UI, as in the on-prem requirements. DNS for those hostnames points at the tenant's CPU nodes, or at a VIP in front of them. * SambaStack 2.1.1 or later, which adds the `global.nodeSelector` setting used to pin the platform to CPU nodes. On the operator's machine: `kubectl`, `helm` v3.19+, and the [vCluster CLI](https://www.vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics). ## Components | Layer | Component | Provided by | Runs on | | :- | :- | :- | :- | | Container runtime, kubelet | kubelet v1.35.10, runc, CNI plugins; containerd v2.1.6 static build | RHEL 8 bundle (step 4), installed by the vCluster join script | Private node | | SELinux policy | `vcluster-selinux` (el8) | vCluster join script, when SELinux is enabled | Private node | | CNI, service proxy, metrics | Flannel, kube-proxy, metrics-server | vCluster (`deploy.*` settings) | Private node | | DNS | CoreDNS | vCluster (`controlPlane.coredns`) | CPU nodes | | Node tunnel | konnectivity agent | vCluster (`controlPlane.advanced.konnectivity`) | Every private node | | Default storage class | local-path-provisioner | vCluster | Private node | | Node role labels | `sambanova.ai/role=control-plane` on CPU nodes, `snRduArch=sn40-16` on RDU nodes | You, per node | Private node | | Tenant control plane | API server, scheduler, controller manager, embedded etcd, konnectivity server | vCluster Platform, `vcluster create` | Host namespace `vcluster-tenant1`, CPU node pool | | Private node licensing | vCluster Platform, free plan | You, once per host | Host | | Tenant app ingress | ingress-nginx | You, per tenant | CPU nodes, host network | | SambaStack platform | `sambastack-base`, `sambastack`: gateway, UI, auth, Postgres, Redis, inference operator | You, per tenant | CPU nodes | | Model bundles | `sambastack-models` | You, per tenant | RDU nodes | ## Set up a tenant Commands run from the operator's machine unless noted. Replace `tenant1`, `vcluster-tenant1`, ``, ``, and `` with your values. ### 1. Install vCluster Platform (once per host) Private nodes aren't in open-source vCluster. They need vCluster Platform. The free plan includes private nodes but is capped at 64 CPUs or 32 GPUs across the instance; check the [vCluster pricing page](https://www.vcluster.com/pricing) against your node sizes. The install must set `config.loftHost` to a hostname that resolves from your machine, from the host nodes, and from pods. ```yaml platform-values.yaml theme={} config: loftHost: ingress: enabled: true host: ingressClass: nginx annotations: nginx.ingress.kubernetes.io/backend-protocol: HTTP nginx.ingress.kubernetes.io/ssl-passthrough: "false" ``` ```shellscript theme={} vcluster platform start --namespace vcluster-platform --email \ --no-tunnel --values platform-values.yaml --version 4.12.4 ``` Then log in to the Platform UI at `https://` as `admin` and activate the free plan. `vcluster platform start` prints the admin password; to read it later, run `helm get values loft -n vcluster-platform` against the host. Until the instance is activated, `vcluster create` fails. Log in from the CLI with the external URL, not a local port-forward, so that tenant control planes can reach vCluster Platform: ```shellscript theme={} vcluster platform login https:// --insecure ``` `--insecure` is for a self-signed Platform certificate. Drop it if the hostname has a trusted certificate. ### 2. Create the tenant control plane The settings that carry the design: | Setting | Value | Why | | :- | :- | :- | | `privateNodes.enabled` | `true` | Worker nodes join the tenant directly. Can't be changed after creation | | `controlPlane.distro.k8s.image.tag` | `v1.35.10` | Pins the tenant to Kubernetes 1.35. The join script downloads the node bundle for this version, and the bundle you build in step 4 must match it | | `controlPlane.service` | `NodePort` 31443 | Exposes the tenant API on every host node IP. Use `LoadBalancer` if your host has one | | `controlPlane.endpoint` | `:31443` | The address the join script and kubelets use | | `controlPlane.proxy.extraSANs` | every host node IP, or the VIP | The API certificate is valid for whichever address the nodes use | | `controlPlane.statefulSet.scheduling.nodeSelector` | `control-plane: shared` | The control plane runs on the host's CPU pool, not on RDU nodes | | `controlPlane.coredns.deployment.nodeSelector` | `sambanova.ai/role: control-plane` | Tenant DNS runs on the tenant's CPU nodes, so RDU maintenance doesn't interrupt name resolution | | `networking.podCIDR`, `serviceCIDR` | tenant-only ranges | Must not overlap with the host or other tenants | | `deploy.cni.flannel`, `kubeProxy`, `localPathProvisioner`, `metricsServer` | `enabled: true` | vCluster installs the node-side basics | ```yaml vcluster.yaml theme={} privateNodes: enabled: true controlPlane: distro: k8s: enabled: true image: tag: v1.35.10 coredns: enabled: true deployment: replicas: 2 nodeSelector: sambanova.ai/role: control-plane backingStore: etcd: embedded: enabled: true statefulSet: persistence: volumeClaim: enabled: true storageClass: openebs-hostpath scheduling: nodeSelector: control-plane: shared service: spec: type: NodePort ports: - name: https port: 443 targetPort: 8443 nodePort: 31443 protocol: TCP endpoint: :31443 proxy: extraSANs: - - - networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 deploy: cni: flannel: enabled: true kubeProxy: enabled: true localPathProvisioner: enabled: true metricsServer: enabled: true ``` If the host's CPU node pool is tainted, add matching `controlPlane.statefulSet.scheduling.tolerations`, or the control plane pod stays `Pending`. CoreDNS stays `Pending` until the first tenant CPU node is joined and labeled in step 6; that is expected. ```shellscript theme={} vcluster create tenant1 -n vcluster-tenant1 -f vcluster.yaml --connect=false --driver helm vcluster connect tenant1 -n vcluster-tenant1 --print \ --server https://:31443 > tenant1.kubeconfig ``` > [!WARNING] > > The kubeconfig holds a cluster-admin client certificate for the tenant. Treat it like a password. ### 3. Prepare the RHEL 8.10 nodes Do this on every CPU node and every RDU node, as root. First confirm cgroup v2; stop here if it prints `tmpfs`: ```shellscript theme={} stat -fc %T /sys/fs/cgroup/ # must print cgroup2fs ``` Stop any existing container runtime, turn off swap, and load the kernel modules and sysctls. The files under `/etc` keep them across reboots: ```shellscript theme={} systemctl disable --now docker.socket docker containerd 2>/dev/null || true swapoff -a sed -i '/\sswap\s/s/^/#/' /etc/fstab cat < /etc/modules-load.d/k8s.conf overlay bridge br_netfilter vxlan nf_conntrack EOF modprobe -a overlay bridge br_netfilter vxlan nf_conntrack cat < /etc/sysctl.d/99-k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system dnf install -y iptables curl tar conntrack-tools socat ``` ### 4. Build the RHEL 8 node bundle Build the bundle once, on any one of the RHEL 8.10 nodes, as root. It replaces the bundle's containerd binaries with containerd's static build of the same version: ```shellscript theme={} cd /root curl -fsSLO https://github.com/loft-sh/kubernetes/releases/download/v1.35.10/kubernetes-v1.35.10-amd64.tar.gz curl -fsSLO https://github.com/containerd/containerd/releases/download/v2.1.6/containerd-static-2.1.6-linux-amd64.tar.gz echo "577900a5a8684c27e344aeeb1fc64e355745f58cba7f83c53649235ba25abbbf containerd-static-2.1.6-linux-amd64.tar.gz" | sha256sum -c - mkdir -p bundle containerd-static tar -xzf kubernetes-v1.35.10-amd64.tar.gz -C bundle tar -xzf containerd-static-2.1.6-linux-amd64.tar.gz -C containerd-static install -m 0755 containerd-static/bin/containerd containerd-static/bin/containerd-shim-runc-v2 \ containerd-static/bin/ctr bundle/release/ file bundle/release/containerd bundle/release/ctr bundle/release/containerd-shim-runc-v2 # each: statically linked bundle/release/containerd --version # containerd ... v2.1.6 tar -czf kubernetes-v1.35.10-amd64-el8.tar.gz -C bundle release ``` Copy `/root/kubernetes-v1.35.10-amd64-el8.tar.gz` to `/root` on every other CPU and RDU node. The bundle version must match the tenant's Kubernetes version (`controlPlane.distro.k8s.image.tag`); build a new bundle when you upgrade the tenant. ### 5. Join the nodes Join the CPU nodes first, then the RDU nodes. Each node joins the same way. From your machine, with the tenant kubeconfig, issue a join token: ```shellscript theme={} KUBECONFIG=tenant1.kubeconfig vcluster token create --expires=4h ``` The command prints a `curl ... | sh -` join command. vCluster prints it with `curl -k`, which skips verification of the tenant API server's certificate. Run it on each node as root, adding `--node-name` and `--bundle-path`: ```shellscript theme={} curl -fsSLk "https://:31443/node/join?token=" | sh -s -- \ --node-name --bundle-path /root/kubernetes-v1.35.10-amd64-el8.tar.gz ``` The join ends with `This node has joined the cluster`. One token joins several nodes until it expires. With SELinux `Enforcing`, the join installs the `vcluster-selinux` package from `rpm.vcluster.com`. On nodes without access to it, install the package first and add `--skip-selinux-rpm` to the join command. ### 6. Label the nodes The labels decide what runs where. CPU nodes get the platform label, and RDU nodes get the RDU label that SambaStack's inference operator selects: ```shellscript theme={} export KUBECONFIG=tenant1.kubeconfig # CPU nodes: SambaStack platform, ingress, DNS kubectl label nodes sambanova.ai/role=control-plane --overwrite # RDU nodes: model bundles only kubectl label nodes snRduArch=sn40-16 --overwrite kubectl get nodes -L sambanova.ai/role,snRduArch ``` Never put both labels on one node. CoreDNS, which was `Pending`, starts on the CPU nodes once they are labeled. ### 7. Install an ingress controller in the tenant Private nodes have no host ingress controller, so the tenant needs its own. This install runs ingress-nginx only on the CPU nodes and binds it to ports 80 and 443 of their host network, so DNS for the tenant's SambaStack hostnames can point at the CPU node IPs. ```shellscript theme={} export KUBECONFIG=tenant1.kubeconfig helm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx --version 4.15.1 \ -n ingress-nginx --create-namespace \ --set controller.kind=DaemonSet --set controller.hostNetwork=true \ --set controller.dnsPolicy=ClusterFirstWithHostNet --set controller.service.type=ClusterIP \ --set controller.ingressClassResource.default=true --set controller.admissionWebhooks.enabled=false \ --set-string 'controller.nodeSelector.sambanova\.ai/role=control-plane' ``` > [!NOTE] > > Any ingress controller that serves the node's ports 80 and 443 and handles standard `Ingress` objects works in place of ingress-nginx. ## Verify From the tenant: ```shellscript theme={} export KUBECONFIG=tenant1.kubeconfig kubectl get nodes -o wide -L sambanova.ai/role,snRduArch # CPU and RDU nodes, Ready, v1.35.10, RHEL 8.10 kubectl -n kube-system get pods -o wide # flannel, kube-proxy, konnectivity agent on every node; CoreDNS on CPU nodes kubectl top node # metrics-server works through the tunnel ``` On the node: ```shellscript theme={} containerd --version # v2.1.6 systemctl is-active containerd kubelet # active, active ``` From the host: ```shellscript theme={} unset KUBECONFIG kubectl get nodes # the private node is not listed kubectl -n vcluster-tenant1 get pods # only the control plane pod, tenant1-0 ``` To check the tunnel, run a test deployment in the tenant and confirm that `kubectl logs` and `kubectl exec` work against it. ## Install SambaStack in the tenant Install SambaStack with the standard [on-prem quickstart](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/getting-started/on-prem.md), using the tenant kubeconfig (`export KUBECONFIG=tenant1.kubeconfig`). SambaNova provides the registry URL, the service-account key and the chart version. Only these settings differ from a normal install. **Node selectors.** The node labels from step 6 decide where each pod runs: * `global.nodeSelector` sends every SambaStack service (gateway, UI, authentication, inference operator, and the rest) to the CPU nodes labeled `sambanova.ai/role=control-plane`. * Postgres, Redis, and OpenEBS come from bundled charts that don't read the global setting, so each gets the same selector in its own section. * Model bundle pods aren't affected. SambaStack's inference operator places them on the RDU nodes labeled `snRduArch=sn40-16`. Add these to `sambastack.yaml`. Requires SambaStack 2.1.1 or later. For the per-service overrides, tolerations, and the other scheduling settings, see the SambaStack Helm values file and the [SambaStack v2.1.1 release notes](https://docs.sambanova.ai/docs/en/release-notes/sambastack.md#sambastack-v2-1-1-release). ```yaml sambastack.yaml (tenant additions) theme={} global: nodeSelector: sambanova.ai/role: control-plane cloudnative-pg: nodeSelector: sambanova.ai/role: control-plane clusterSpec: affinity: nodeSelector: sambanova.ai/role: control-plane global-queue-redis: master: nodeSelector: sambanova.ai/role: control-plane response-queue-redis: master: nodeSelector: sambanova.ai/role: control-plane openebs: localpv-provisioner: localpv: nodeSelector: sambanova.ai/role: control-plane ``` Skip the quickstart's node labeling step; the nodes were labeled in step 6. **Storage classes.** SambaStack installs its own OpenEBS provisioner and stores Postgres data in the `openebs-hostpath` class, under `/scratch/openebs`. With the selectors above, that path is on the CPU nodes, so give them enough disk there, or set `openebs.localpv-provisioner.hostpathClass.basePath`. **Ingress and DNS.** Use the ingress-nginx controller from step 7, which is the tenant's default ingress class. Point the DNS records for the SambaStack API and UI hostnames at the CPU node IPs, or a VIP in front of them, never at RDU nodes. The hostnames must also resolve from inside the tenant, and the TLS certificate must come from a trusted CA, as the quickstart requires. After the install, check that the platform pods are on CPU nodes and the model pods on RDU nodes: ```shellscript theme={} kubectl -n sambastack get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName | sort -k2 kubectl get nodes -L sambanova.ai/role,snRduArch ``` From the host (with `KUBECONFIG` unset), nothing changes: `kubectl -n vcluster-tenant1 get pods` still shows only the control plane pod. ## Maintain an RDU node Because the platform runs on CPU nodes, RDU maintenance only affects the model replicas on that node. Plan at least two RDU nodes per model if it must keep serving during maintenance. ```shellscript theme={} export KUBECONFIG=tenant1.kubeconfig # Stop new pods landing on the node and move the model pods off it kubectl cordon kubectl drain --ignore-daemonsets --delete-emptydir-data # ... firmware, OS, or RDU runtime work, reboot ... kubectl uncordon ``` The gateway, UI, login, Postgres, Redis, and DNS keep running throughout. Drain CPU nodes one at a time, and only while the others are `Ready`. ## Limitations * This reference is tested with RHEL 8.10 private nodes only. It isn't tested with RHEL 10 nodes. * RHEL 8.10 nodes need the RHEL 8 bundle from step 4. Rebuild it for every tenant Kubernetes upgrade, and validate it on a test node first. * vCluster's node-to-node VPN isn't available on RHEL 8.10 nodes. This reference doesn't use it. * The join token expires. Issue a new one for each additional node. * The tenant API sits on one NodePort of the host nodes. Use a `LoadBalancer` service or a VIP for production. * vCluster Platform must reach `admin.loft.sh` to validate the free plan, and the free plan's CPU and GPU caps apply. * Tenant control plane state is on one volume. For production, back it up or run vCluster's high-availability configuration. ## Cleanup ```shellscript theme={} # Tenant: remove SambaStack and the ingress controller export KUBECONFIG=tenant1.kubeconfig helm uninstall sambastack-models sambastack sambastack-base -n sambastack helm uninstall ingress-nginx -n ingress-nginx # Node, as root: remove the node components (needs a fresh token) curl -fsSLk "https://:31443/node/join?token=" | sh -s -- --reset-only rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/99-k8s.conf # Host: remove the tenant unset KUBECONFIG vcluster delete tenant1 -n vcluster-tenant1 ``` Platform stays for other tenants. To remove it as well: `helm uninstall loft -n vcluster-platform`. ## Additional resources * [vCluster: private nodes](https://www.vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/) * [vCluster: join private nodes](https://www.vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join) * [vCluster: private node requirements](https://www.vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements) * [vCluster production guide: choose a worker node model](https://www.vcluster.com/docs/vcluster/production-guide/choose-worker-node-model) * [containerd releases](https://github.com/containerd/containerd/releases) * [ingress-nginx Helm chart](https://kubernetes.github.io/ingress-nginx/deploy/)