Deploy a Model Bundle on SambaStack - SambaNova Documentation

This page describes how to serve a bundle by referencing it from a ModelDeployment. It applies both to the bundles that SambaNova provides and to bundles you authored yourself. To author your own bundle first, see Create a custom bundle. To serve a single model without a bundle, see Deploy a single model.

Prerequisites

Before deploying, complete the quickstart that applies to you:

Quickstart - Hosted

System set up for hosted SambaStack

Quickstart - On-prem

System set up for on-prem SambaStack

Additionally:

Review the ModelDeployment field reference.
Confirm the bundle you intend to deploy has passed validation. See Legalizer results.
Review Supported models and bundles for the bundles SambaNova provides.

Deploy the bundle

Discover the available bundles

List the bundles present in the cluster and note the name of the one you want to deploy:

Hosted
On Premise

kubectl get modelbundles

kubectl -n <namespace> get modelbundles.sambanova.ai

Output:

mb-gpt120                      17h
mb-gpt120-llama70sd8-llama8    17h
ds-v32-gemma-4-31b             3h

Before deploying, confirm the bundle passed validation, as described in Legalizer results.

Create a ModelDeployment

Insert the bundle name into spec.bundle:

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120
spec:
  bundle: mb-gpt120
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader

Apply the ModelDeployment

Hosted
On Premise

kubectl apply -f <modeldeployment-file>.yaml

kubectl -n <namespace> apply -f <modeldeployment-file>.yaml

Monitor deployment status

Hosted
On Premise

kubectl get modeldeployments
kubectl describe modeldeployment <deployment-name>

kubectl -n <namespace> get modeldeployments.sambanova.ai
kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>

Verify the deployment

Pods reporting ready means the serving containers started, not that the model answers requests. Confirm the serving name, then send one inference request.

Resolve the serving name

The name you send to the API is not the Kubernetes resource name. Three names are involved:

Name	Where it lives	Used for
`metadata.name` on a Model	The `Model` resource	`kubectl` commands, and `modelConfigs[].model` references in bundles and deployments
`spec.name` on a Model	The `Model` resource	The `model` field of an inference request
`spec.aliases` on a Model	The `Model` resource	Additional names that route to the same model

Read the serving name from the Model you deployed:

Hosted
On Premise

kubectl get model <model-resource-name> -o jsonpath='{.spec.name}{"\n"}'

kubectl -n <namespace> get model.sambanova.ai <model-resource-name> -o jsonpath='{.spec.name}{"\n"}'

Then confirm the gateway is serving it. Each id in the response is a servable name:

curl https://<your-api-domain>/v1/models \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY"

A model set to modelSettings.routable: false, such as a speculative decoding draft model, is deliberately absent from this list and cannot be addressed directly.

Send a request

Use the serving name in the model field:

curl https://<your-api-domain>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Say hi in one sentence."}
    ],
    "max_tokens": 128,
    "stream": false
  }'

A successful response returns the serving name in model and a populated choices[0].message:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Hi there, it's a sunny day today!",
        "role": "assistant"
      }
    }
  ],
  "created": 1770927918,
  "id": "c754b468-05e1-4fe4-8a28-f51ecc74f4f2",
  "model": "gpt-oss-120b",
  "object": "chat.completion",
  "system_fingerprint": "fastcoe",
  "usage": {
    "completion_tokens": 121,
    "prompt_tokens": 91,
    "total_tokens": 212,
    "stop_reason": "stop",
    "time_to_first_token": 0.05286884307861328,
    "total_latency": 0.29257988929748535
  }
}

The usage object carries additional throughput and latency fields; for the full body, and for the API domain to use in each environment, see Quickstart - Hosted or Quickstart - On-prem.

A model not found error with pods running usually means the request used metadata.name instead of spec.name. An inference error on a model that loads and lists usually means the checkpoint and the profile disagree on model_arch. See Deployment failures.

Switch bundles

A ModelDeployment’s bundle reference cannot be changed in place. To serve a different bundle, delete the existing deployment and apply a new one that references the new bundle.

Delete the existing ModelDeployment

Hosted
On Premise

kubectl delete modeldeployment <deployment-name>

kubectl -n <namespace> delete modeldeployment <deployment-name>

Apply a ModelDeployment for the new bundle

Author a new ModelDeployment whose spec.bundle names the bundle you want to serve, then apply it as described in Deploy the bundle.

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-qwen3-32b-whisper
spec:
  bundle: mb-qwen3-32b-whisper
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader

To request new bundles, contact SambaNova support.

Deploy multiple bundles

To serve more than one bundle at a time, apply a separate ModelDeployment for each bundle.

SambaStack supports only one bundle per node. When deploying multiple bundles, assign each bundle to separate nodes to avoid resource conflicts. Use spec.nodeSelector on each ModelDeployment to control placement.

Verify that the pods reflect the deployed bundles:

Hosted
On Premise

kubectl get pods

kubectl -n <namespace> get pods

Update or remove a deployment or bundle

Update a configuration
Remove a configuration

Modify the YAML file

Edit the ModelBundle, ModelProfile, or ModelDeployment YAML file with your changes.

Reapply the configuration

Hosted
On Premise

kubectl apply -f <modified-file>.yaml

kubectl -n <namespace> apply -f <modified-file>.yaml

The legalizer automatically revalidates the changes. Changing a profile or a model triggers a reconcile of every bundle and deployment that references it.

Delete the ModelDeployment

Hosted
On Premise

kubectl delete modeldeployment <deployment-name>

kubectl -n <namespace> delete modeldeployment <deployment-name>

Delete the ModelBundle (if one was used)

Hosted
On Premise

kubectl delete modelbundle <bundle-name>

kubectl -n <namespace> delete modelbundle <bundle-name>

Delete the ModelProfile (optional)

Delete a profile only if you authored it for a custom architecture. Profiles provided with the SambaStack installation are shared across models and should be left in place.

Hosted
On Premise

kubectl delete modelprofile <profile-name>

kubectl -n <namespace> delete modelprofile <profile-name>

Troubleshooting

For the failures the operator reports once it starts creating serving pods, and for the failures the legalizer reports before that, see Troubleshooting deployments.

Supported models and bundles

Catalogue of models and bundles available for deployment

ModelDeployment

Full field reference for the deployment resource

Deploying custom checkpoints

Deploy your own custom or fine-tuned checkpoints

Migrating bundle configuration

Migrate bundles built with the deprecated custom resources

Get started

Models

Features

Build

Resources

Deploy a bundle

Prerequisites

Quickstart - Hosted

Quickstart - On-prem

Deploy the bundle

Verify the deployment

Resolve the serving name

Send a request

Switch bundles

Deploy multiple bundles

Update or remove a deployment or bundle

Troubleshooting

Supported models and bundles

ModelDeployment

Deploying custom checkpoints

Migrating bundle configuration

​Prerequisites

Quickstart - Hosted

Quickstart - On-prem

​Deploy the bundle

​Verify the deployment

​Resolve the serving name

​Send a request

​Switch bundles

​Deploy multiple bundles

​Update or remove a deployment or bundle

​Troubleshooting

​Related documentation

Supported models and bundles

ModelDeployment

Deploying custom checkpoints

Migrating bundle configuration

Prerequisites

Deploy the bundle

Verify the deployment

Resolve the serving name

Send a request

Switch bundles

Deploy multiple bundles

Update or remove a deployment or bundle

Troubleshooting

Related documentation