Quickstart - On-Prem - SambaNova Documentation

This guide walks you through the basic steps to set up SambaStack on-prem, including authentication, namespace creation, secrets configuration, and Helm deployment.

SambaStack on-prem architecture

Diagram of SambaStack on-premises deployment architecture

The diagram shows five connection flows: Inference requests

Users and developers send requests through a Load Balancer to the Datacenter Network
The Datacenter Network routes traffic to the SambaRack Cluster over a 100G/400G front-end network
The front-end network delivers requests to the Core Services Rack and the SambaRack SN40L-16 units, which run inference

Administration

Administrators reach the cluster over two separate networks: the front-end network for cluster operations, and a dedicated 1Gbps management network for out-of-band hardware access
The SMS node and the SambaRack SN40L-16 units both also connect to the management network

Supporting infrastructure

Storage and three customer-managed services – AUTH (OIDC), DNS, and NTP – connect into the Datacenter Network

Model and software delivery

The SambaNova artifact repository (hosted on GCP) delivers checkpoints and software updates directly to the CP nodes – this bypasses the Datacenter Network entirely, it’s a separate, direct path

Internal cluster communication

SambaRack SN40L-16 units communicate with each other over a dedicated SambaRack Cluster Private network

Prerequisites

This section outlines the prerequisites for deploying SambaStack and guides you through obtaining the necessary credentials and configurations.

SambaNova collects hardware prerequisites (such as load balancer configuration) and OS configuration requirements (such as NTP servers) through the SambaNova site survey. Contact your SambaNova representative for details.

SambaRack SN40L-16 systems may be deployed with either 12TB or 4TB of DDR memory, and mixed-memory clusters are supported when the cluster is configured to use the lower memory limit across all nodes. All newly shipped SN40L-16 hardware is configured with 4TB of DDR memory, while existing 12TB systems can continue operating at their full capacity when deployed in uniform clusters. SambaNova-provided bundles are validated for 4TB systems, and SambaStack also supports cluster-level memory limit enforcement to ensure bundle configurations stay within the declared node capacity.

This guide assumes you have a kubernetes cluster already in place.
You must provide two TLS certificates and DNS names for the API and UI endpoints:
- api.<your-domain> – certificate and private key
- ui.<your-domain> – certificate and private key
You must provide certificates as either:
- Two individual certificates (one for each hostname), or
- A single wildcard certificate (e.g., *.example.com) covering both
Self-signed certificates are not supported for production environments

Certificates must be issued by a trusted Certificate Authority (CA) such as Let’s Encrypt, DigiCert, or your organization’s internal CA

Ensure the following tools are installed/present:

Requirement	Specification
Kubernetes version	v1.30+ (recommended)
Helm version	v3.19+ (recommended)
Control planes	Core Services Rack (CSR) includes 3 control planes for SambaStack
Data planes	SambaRack worker nodes serve as data planes

Make sure you have received the following from SambaNova
1. Helm chart OCI registry URLs
  Chart Registry URL
  SambaStack Base oci://<REGISTRY_URL>/sambastack/sambastack-base
  SambaStack oci://<REGISTRY_URL>/sambastack/sambastack
  
  SambaNova provides the full registry URL, service-account-key file, and version number during handover. Contact your SambaNova representative for access credentials.
  
  Version numbers change with new chart releases. Check with your SambaNova representative for the current installation version.
2. Configuration files
  - Sample sambastack.yaml file for your deployment - this guide also shows how to build this from scratch.
3. Google Cloud service account key file. This file is used to create:
  - Image Pull Secret
  - Artifact reader secret

Chart	Registry URL
SambaStack Base	`oci://<REGISTRY_URL>/sambastack/sambastack-base`
SambaStack	`oci://<REGISTRY_URL>/sambastack/sambastack`

Setup

Authenticate and Configure Cluster

Create sambastack namespace
kubectl create namespace sambastack

Create a Docker registry secret for pulling images from Artifact Registry using your Google Cloud service account JSON key:

kubectl create secret docker-registry regcred \
  --docker-server=us-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat <service-account-key>.json)" \
  --namespace sambastack

Create a secret for reading model artifacts from Google Cloud Storage:

kubectl create secret generic sambanova-artifact-reader \
  --from-file=GOOGLE_APPLICATION_CREDENTIALS=<service-account-key>.json \
  --namespace sambastack

Label each RDU node for scheduling. Add --overwrite for idempotency:
kubectl label nodes <NODE_NAME> snRduArch=sn40-16 --overwrite
Repeat this command for each node in your cluster.

Configure `sambastack.yaml`

Configure a minimum viable sambastack.yaml file:

gateway:
  replicas: 3
  ingress:
    hosts:
      - host: api.<yourdomain>
        tlsSecretName: tls-cert
cloud-ui:
  replicas: 3
  ingress:
    hosts:
      - host: ui.<yourdomain>
        tlsSecretName: tls-cert
bundles:
  bundleSpecs:
    - name: gpt-oss-120b-8-32-64-128k
  bundleDeploymentSpecs:
    - name: gpt-oss-120b-8-32-64-128k
      groups:
        - name: default
          minReplicas: 1
          qosList: [web, free]
global:
  extraEnv:
   - name: "DDR_PER_RDU_GB"
     value: "256"

This default file deploys a bundle that you will be able to query by including entries under the bundles.bundleSpecs and bundles.bundleDeploymentSpecs sections.

SambaRack SN40L-16 systems may have either 12TB or 4TB of DDR memory, but all newly shipped systems are 4TB, so configure the cluster memory limit accordingly and use the lower limit when mixing node types.

See the SambaStack.yaml Reference for a full example.

Helm Login and Install

Use your Google Cloud service account to authenticate with the Helm registry:

export GOOGLE_APPLICATION_CREDENTIALS="<path/to/sa.json>"
gcloud auth activate-service-account --key-file="$GOOGLE_APPLICATION_CREDENTIALS"
helm registry login -u oauth2accesstoken -p "$(gcloud auth print-access-token)" us-docker.pkg.dev

Install the base chart and main SambaStack chart:

helm upgrade \
  --install \
  --namespace sambastack \
  --version <NEW_VERSION> \
  sambastack-base \
  oci://<REGISTRY_URL>/sambastack/sambastack-base
helm upgrade \
  --install \
  --namespace sambastack \
  --version <NEW_VERSION> \
  sambastack \
  -f sambastack.yaml \
  oci://<REGISTRY_URL>/sambastack/sambastack

SambaNova provides the full registry URL and version number during handover. Contact your SambaNova representative for access credentials.

Version numbers change with new chart releases. Use the version number provided by your SambaNova representative.

Refer to the Pods reference table for details on the pods created after installation.

Ensure Pods are Running

Run the following command to see all the pods running in your cluster:
kubectl -n sambastack get pods

Make sure all pods are Running. Your response should resemble the following:

NAME                                              READY   STATUS      RESTARTS        AGE
auth-and-billing-5f7f9cc8db-4mlnx                 2/2     Running     0               20m
auth-and-billing-5f7f9cc8db-lmts8                 2/2     Running     0               20m
auth-and-billing-5f7f9cc8db-rztfp                 2/2     Running     0               20m
cloud-ui-76d458db46-2k2ws                         1/1     Running     0               20m
cloud-ui-76d458db46-2z7jd                         1/1     Running     0               20m
cloud-ui-76d458db46-rsx4p                         1/1     Running     0               20m
db-admin-5b8c7fc5b4-5lvl4                         2/2     Running     0               20m
debug-pod                                         1/1     Running     0               20m
gateway-7464677785-7kpvf                          4/4     Running     0               20m
gateway-7464677785-cm47v                          4/4     Running     0               20m
gateway-7464677785-kntm8                          4/4     Running     0               20m
inf-gpt-oss-120b-8-32-64-128k-cache-0             1/1     Running     0               20m
inf-gpt-oss-120b-8-32-64-128k-q-default-n-0       2/2     Running     0               20m
inf-operator-0                                    1/1     Running     0               20m
inference-router-77c7cccd5b-dkxzc                 1/1     Running     0               20m
keycloak-0                                        1/1     Running     0               20m
keycloak-operator-574d46bf79-wh8fr                1/1     Running     0               20m
sambastack-cloudnative-pg-cff97c7c5-fxznp         1/1     Running     0               20m
sambastack-global-queue-redis-master-0            2/2     Running     0               20m
sambastack-localpv-provisioner-77558bf687-dsqnr   1/1     Running     0               20m
sambastack-postgres-1                             1/1     Running     0               20m
sambastack-postgres-2-join-l7scz                  1/1     Running     0               20m
sambastack-response-queue-redis-master-0          2/2     Running     0               20m

It usually takes about 5-10 minutes for all the pods to reach their desired Running state.

See the Pods reference table for more details.

Create User and API Key

This quickstart guide uses the default keycloak auth setup to get started. To configure custom authentication, check out the authentication page.

Retrieve keycloak temp-admin credentials:

kubectl -n sambastack get secret keycloak-initial-admin -o go-template='username: {{.data.username | base64decode}} password: {{.data.password | base64decode}}'

Port forward keycloak

kubectl -n sambastack port-forward svc/keycloak-service 8080

Log into keycloak at http://localhost:8080/ with temp-admin credentials above.
Create admin user
Assign admin role
Set admin credentials
Add admin user email to the db-admin section of sambastack.yaml:
db-admin: admins: - temp-admin@cluster.local - <admin-user-email>
See the SambaStack.yaml Reference for a full example.
Visit ui.<your-domain> and log in with user credentials retrieved above.
Create API key and save securely:

Query Deployed Model Bundle

Set your API key:

export SAMBANOVA_API_KEY='<API key from step 5>'

Run a test query against the deployed model bundle with either cURL or a python script:

cURL
Python

Run the following cURL command

curl https://api.<your-domain>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Say hi in one sentence and comment on the weather."}
    ],
    "temperature": 0.2,
    "max_tokens": 128,
    "stream": false
  }'

Your response should resemble the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "Hi there, it's a sunny day today!",
		"reasoning": "The user wants a short response: \"Say hi in one sentence and comment on the weather.\" So we need to produce a single sentence that says hi and comments on the weather. Must be concise. So something like \"Hi there! It's a sunny day today.\" That's two sentences. Need one sentence. Could be \"Hi there–it's a sunny day today!\" That's one sentence with dash. Or \"Hi there, it's a sunny day today!\" That's one sentence with comma. That works. Provide that.",
        "role": "assistant"
      }
    }
  ],
  "created": 1770927918,
  "id": "c754b468-05e1-4fe4-8a28-f51ecc74f4f2",
  "model": "gpt-oss-120b",
  "object": "chat.completion",
  "system_fingerprint": "fastcoe",
  "usage": {
    "completion_tokens": 121,
	"completion_tokens_after_first_per_sec": 500.6027126944832,
    "completion_tokens_after_first_per_sec_first_ten": 502.05732024083596,
    "completion_tokens_after_first_per_sec_graph": 502.05732024083596,
    "completion_tokens_per_sec": 413.56225915094006,
    "end_time": 1770927918.2348678,
    "is_last_response": true,
    "prompt_tokens": 91,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "start_time": 1770927917.942288,
    "stop_reason": "stop",
    "time_to_first_token": 0.05286884307861328,
    "time_to_first_token_graph": 0.05082893371582031,
    "total_latency": 0.29257988929748535,
    "total_tokens": 212,
    "total_tokens_per_sec": 724.5884209917298
  }
}

Create a python script to run the following program. Make sure to insert your correct api domain name.

import os
from sambanova import SambaNova

api_key = os.environ.get("SAMBANOVA_API_KEY")

client = SambaNova(
    api_key=api_key,
    base_url="https://api.<your-domain>/v1",
)

try:
    response = client.chat.completions.create(
        model="gpt-oss-120b",
        messages=[
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "Say hi in one sentence and comment on the weather."}
        ],
        temperature=0.2,
    )
    print(response.choices[0].message.content)

except Exception as e:
    print(f"An error occurred: {e}")

Your response should resemble the following:

Hi, it's a sunny day today!

Next steps

Congrats! You’ve set up your SambaStack cluster to a baseline running state. At this point, what you configure next depends on your goals. The following is a list of common configurations you can set up next:

​SambaStack on-prem architecture

​Prerequisites

​Setup

Authenticate and Configure Cluster

Configure `sambastack.yaml`

Helm Login and Install

Ensure Pods are Running

​Next steps

SambaStack on-prem architecture

Prerequisites

Setup

Next steps