Deploying Models and Bundles on SambaStack

SambaStack manages model serving through a set of Kubernetes custom resources. This page covers how those resources relate to one another, how to inspect the ones already present in your cluster, and which workflow to follow for the deployment you want. The pages in this section walk through each workflow; the field reference for an individual resource lives on its own page under Custom resources. In SambaStack, a deployment serves one or more models on a single node, together with their configurations, including batch sizes and sequence lengths. This approach uses the SambaNova Reconfigurable Dataflow Unit (RDU) to support multiple models and configurations in a single deployment. Because every model in a deployment is already resident, switching between them does not require reloading weights, which increases throughput compared with deployments that load a single static model. For example, a single bundle can serve both Llama-3.3-70B-Instruct and Llama-3.1-8B-Instruct, allowing you to switch between them almost instantly. Each configuration occupies space on the node, so different bundles contain different sets of configurations. A copy of Llama-3.3-70B-Instruct with a batch size of 4 and a sequence length of 16k represents one configuration. For the bundles that SambaNova provides, see Supported models and bundles.

SambaStack v2.0.2 introduces a new set of Kubernetes custom resources (CRs) to manage how bundles are deployed. The CRs in older versions of SambaStack are deprecated; however, they remain functional until September 30, 2026. Model bundles built using the deprecated CRs will need to be migrated to the new resource model.

A bundle is optional. To serve a single model, pair it with a model profile and deploy it directly. Create a bundle when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration.

Terminology

Term	Definition
RDU	Reconfigurable Dataflow Unit - SambaNova’s proprietary processor architecture
PEF	Processor Executable Format - Compiled model binaries that run on RDUs
Bundle	A deployable package combining models, their profiles, and their configurations
Configuration	The runtime settings for a model, including batch size and sequence length. A single deployment can include multiple configurations, enabling instant switching between them
Model architecture	The `model_arch` value shared by a set of PEFs, which determines the checkpoints those PEFs are compatible with
Expert	A sequence length profile configuration (for example, 8k, 16k, 32k) within a model
Tier	A sequence length key in a batching configuration (for example, `8k` or `128k`), which maps to the batch sizes served at that length
Speculative Decoding	An optimization technique using a smaller draft model to accelerate inference from a larger target model
Legalizer	A validation process that verifies a configuration fits within RDU memory constraints
CR (Custom Resource)	A Kubernetes extension that defines custom resource types such as `Pef`, `Model`, `ModelProfile`, `ModelDeployment`, and `ModelBundle`

Resource architecture

This section covers the resource structure used when creating custom deployments. All of these resources use apiVersion: sambanova.ai/v1alpha1. Every node is tagged with its role. Nodes tagged provided have dashed borders; they come with the SambaStack installation and you reference them by name. Nodes tagged you author have solid borders; they are the resources you write. The rounded node tagged not a CR is not a custom resource at all, but a single Model and ModelProfile pairing, which corresponds to one entry in spec.modelConfigs. The resources divide responsibility as follows:

Pef - Registers a compiled executable and the metadata describing what it supports, including its model architecture, batch size, and maximum sequence length.
ModelProfile - Defines how a model architecture can be run: which PEFs it uses, which features it supports, and the batch sizes available at each sequence length tier.
Model - Defines which checkpoints are available for each architecture, along with the tokenizer to use.
ModelBundle - Combines one or more model and profile pairs into a single named, deployable unit. Optional.
ModelDeployment - Instantiates one or more replicas on the cluster, either from a bundle or from an inline model and profile pair.

This separation allows you to:

Deploy a single model by pairing it with a profile, without creating a bundle
Reuse one profile across every model and checkpoint that shares its architecture, including custom checkpoints for fine-tuned models
Serve a custom checkpoint by adding a Model resource only, leaving the profile untouched
Group several models into one validated, shareable unit when you need them served together
Update checkpoints without modifying profiles or deployment configurations
Deploy the same configuration with different replica counts

Checkpoint paths are never declared in a bundle or a deployment. The operator resolves them from the Model you reference, either through the <model-name>[:<arch>][:<version>] reference or through modelSettings.checkpointOverrides.Compatibility between a checkpoint and a profile is defined by model_arch, but the operator does not verify it. Select a checkpoint whose architecture is compatible with the profile you reference.

Identify available models and profiles

Before authoring a deployment, identify the models and profiles available in your cluster and determine which pair together.

List available models

List the models and note the architecture keys under each model’s spec.checkpoints:

Hosted
On Premise

kubectl get models

Example:

kubectl describe model gpt-oss-120b

kubectl -n <namespace> get models.sambanova.ai

Example:

kubectl -n <namespace> describe model.sambanova.ai gpt-oss-120b

These commands list Kubernetes resource names (metadata.name), which are not the names you send to the inference API. Use the serving name from spec.name in requests. See Supported models and bundles.

List available profiles

List the profiles and note the model_arch value of each:

Hosted
On Premise

kubectl get modelprofiles

kubectl -n <namespace> get modelprofiles.sambanova.ai

Output:

deepseek              17h
gpt-oss-fp8-dyt       17h
llama-3p1-70b-sd      17h
llama-3p1-8b          17h

Match a profile to a model

View a profile to confirm its architecture, features, and batching support:

Hosted
On Premise

kubectl describe modelprofile <profile-name>

kubectl -n <namespace> describe modelprofile.sambanova.ai <profile-name>

A profile can be used with a model when the profile’s model_arch value matches one of the architecture keys under the model’s spec.checkpoints. The operator does not verify this pairing, so confirm it yourself before deploying. Review the profile for:

model_arch - The architecture the profile targets
features - Optional features the profile’s PEFs support
batchingConfigs - Named batching configurations, each mapping sequence length tiers to their batch sizes
pefs - The PEF versions the profile references

A profile whose features list includes speculative_decoding requires a draft model. Pair it using spec.specDecodingPairs in a bundle.

Choose a workflow

Deploy a single model

Pair a model with a profile and deploy it directly, without a bundle

Deploy a bundle

Serve a bundle that SambaNova provides, or one you authored

Create a custom bundle

Combine several models and profiles into one named, validated unit

Deploying custom checkpoints

Convert, upload, and serve your own custom or fine-tuned checkpoint

High-throughput vs. high-interactivity

Choose between the two dedicated deployment configurations

Migrating bundle configuration

Migrate bundles built with the deprecated custom resources

Get started

Models

Features

Build

Resources

Deploying models and bundles

Terminology

Resource architecture

Identify available models and profiles

Choose a workflow

Deploy a single model

Deploy a bundle

Create a custom bundle

Deploying custom checkpoints

High-throughput vs. high-interactivity

Migrating bundle configuration

​Terminology

​Resource architecture

​Identify available models and profiles

​Choose a workflow

Deploy a single model

Deploy a bundle

Create a custom bundle

Deploying custom checkpoints

High-throughput vs. high-interactivity

Migrating bundle configuration

Terminology

Resource architecture

Identify available models and profiles

Choose a workflow