Deploying Models and Bundles

This guide describes how to deploy models with specific configurations on SambaStack by authoring custom resources: a ModelDeployment to serve a single model, or a ModelBundle to serve several models together. In SambaStack, a deployment serves one or more models on a single node, together with their configurations, including batch sizes and sequence lengths. This approach uses the SambaNova Reconfigurable Dataflow Unit (RDU) to support multiple models and configurations in a single deployment. Because every model in a deployment is already resident, switching between them does not require reloading weights, which increases throughput compared with deployments that load a single static model. For example, a single bundle can serve both Llama-3.3-70B-Instruct and Llama-3.1-8B-Instruct, allowing you to switch between them almost instantly. Each configuration occupies space on the node, so different bundles contain different sets of configurations. A copy of Llama-3.3-70B-Instruct with a batch size of 4 and a sequence length of 16k represents one configuration. For the bundles that SambaNova provides, see Supported Models and Bundles.

SambaStack v2.0.2 introduces a new set of Kubernetes custom resources (CRs) to manage how bundles are deployed. The CRs in older versions of SambaStack are deprecated and will be functional until September 30, 2026. Model bundles built using the deprecated CRs will need to be migrated.

A bundle is optional. To serve a single model, pair it with a model profile and deploy it directly. Create a bundle when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration.

This guide covers both deploying the bundles that SambaNova provides and authoring your own. For the catalogue of provided bundles, see Supported Models and Bundles.

Prerequisites

Before creating custom deployments or bundles, complete the following that applies to you:

Quickstart - Hosted

System set up for hosted SambaStack

Quickstart - On-prem

System set up for On-prem Sambastack

Additionally, review the following documentation:

Supported Models and Bundles - Available model checkpoints
Speculative Decoding Deployment Guidelines - Required if configuring speculative decoding pairs

Terminology

Term	Definition
RDU	Reconfigurable Dataflow Unit - SambaNova’s proprietary processor architecture
PEF	Processor Executable Format - Compiled model binaries that run on RDUs
Bundle	A deployable package combining models, their profiles, and their configurations
Configuration	The runtime settings for a model, including batch size and sequence length. A single deployment can include multiple configurations, enabling instant switching between them
Model architecture	The `model_arch` value shared by a set of PEFs, which determines the checkpoints those PEFs are compatible with
Expert	A sequence length profile configuration (for example, 8k, 16k, 32k) within a model
Tier	A sequence length key in a batching configuration (for example, `8k` or `128k`), which maps to the batch sizes served at that length
Speculative Decoding	An optimization technique using a smaller draft model to accelerate inference from a larger target model
Legalizer	A validation process that verifies a configuration fits within RDU memory constraints
CR (Custom Resource)	A Kubernetes extension that defines custom resource types such as `Pef`, `Model`, `ModelProfile`, `ModelDeployment`, and `ModelBundle`

Concepts

Resource architecture

This section covers the resource structure used when creating custom deployments. All of these resources use apiVersion: sambanova.ai/v1alpha1. Light boxes are provided with the SambaStack installation and referenced by name; darker boxes are what you author. The amber box is not a custom resource, but a single Model and ModelProfile pairing, which corresponds to one entry in spec.modelConfigs. The resources divide responsibility as follows:

Pef - Registers a compiled executable and the metadata describing what it supports, including its model architecture, batch size, and maximum sequence length.
ModelProfile - Defines how a model architecture can be run: which PEFs it uses, which features it supports, and the batch sizes available at each sequence length tier.
Model - Defines which checkpoints are available for each architecture, along with the tokenizer to use.
ModelBundle - Combines one or more model and profile pairs into a single named, deployable unit. Optional.
ModelDeployment - Instantiates one or more replicas on the cluster, either from a bundle or from an inline model and profile pair.

This separation allows you to:

Deploy a single model by pairing it with a profile, without creating a bundle
Reuse one profile across every model and checkpoint that shares its architecture, including custom checkpoints for fine-tuned models
Serve a custom checkpoint by adding a Model resource only, leaving the profile untouched
Group several models into one validated, shareable unit when you need them served together
Update checkpoints without modifying profiles or deployment configurations
Deploy the same configuration with different replica counts

Checkpoint paths are never declared in a bundle or a deployment. The operator resolves them from the Model you reference, either through the <model-name>[:<arch>][:<version>] reference or through modelSettings.checkpointOverrides.Compatibility between a checkpoint and a profile is defined by model_arch, but the operator does not verify it. Select a checkpoint whose architecture is compatible with the profile you reference.

ModelProfile structure

A ModelProfile defines the runtime shape of a single model architecture. Profiles are reusable, and the profiles for supported architectures are provided with the SambaStack installation, so in most cases you reference an existing profile by name rather than authoring one.

apiVersion: sambanova.ai/v1alpha1
kind: ModelProfile
metadata:
  name: gpt-oss-fp8-dyt
spec:
  model_arch: gpt-oss-fp8
  features: []
  defaultBatchingConfig:
    8k:
      batch_sizes: [2, 4, 6, 8]
    32k:
      batch_sizes: [2, 4, 6, 8]
    64k:
      batch_sizes: [2, 4]
    128k:
      batch_sizes: [2]
  pefs:
    - gpt-oss-fp8-ss131072-bs8-dyt-1:1
  secretNames:
    - sambanova-artifact-reader

ModelProfile Top-Level Fields

Field	Required	Description
`spec.pefs`	Yes	Ordered list of PEF custom resource references, each in `<pef-name>[:<version>]` format. The list must be non-empty and must not contain duplicate PEF names. If the version suffix is omitted, the latest version is used.
`spec.model_arch`	Yes	Model architecture targeted by this profile. All PEFs in a profile share the same `model_arch`, and this field makes that value visible without inspecting the individual `Pef` resources.
`spec.features`	Yes	Features supported by this profile, derived from its PEFs. Known values are `continuous_batching`, `prompt_caching`, `speculative_decoding`, `speculative_prefill`, and `constrained_decoding`. An empty list means no optional features.
`spec.defaultBatchingConfig`	Yes	Default batching support, keyed by sequence length tier. See Batching configuration. If omitted, the operator generates the configuration from the PEF grouping.
`spec.secretNames`	Yes	List of Kubernetes secrets required in order to download the PEFs. Must match secrets configured in your environment.
`spec.skip_legalizer`	No	Skips legalizer validation for this profile.
`status.batchingConfig`	N/A	Read-only. The resolved default batching configuration, taken from the spec or generated by the operator.

Batching configuration

A batching configuration maps each sequence length tier to the batch sizes served at that length. Common tier keys are 8k, 16k, 32k, 64k, and 128k. PEFs below 1k use a bare integer key, for example 448, and vocoder PEFs use a codes-length key in the form <n>t, for example 10t.

spec:
  defaultBatchingConfig:
    128k:
      batch_sizes: [1]
    32k:
      batch_sizes: [1, 2, 4]
    8k:
      batch_sizes: [1, 2, 4, 8]
    4k:
      batch_sizes: [1, 2, 4, 8]
      is_default: true

Parameter	Required	Description
`batch_sizes`	Yes	In a `defaultBatchingConfig`, an explicit list of integers, for example `[1, 2, 4]`. In an override, this can also be the string `"*"`. See Overriding a batching configuration.
`is_default`	No	If `true`, this tier is served as the `default` expert. At most one tier per batching configuration may set this.

Including multiple batch sizes for each tier allows the inference engine to select the smallest and fastest configuration based on current workload.

Overriding a batching configuration

A ModelProfile establishes the batch sizes available at each tier, so a defaultBatchingConfig always lists them explicitly. It never uses "*". The tier-to-batch_sizes structure is reused when you override a profile’s defaults for one model, either through modelConfigs[].batchingConfig in a ModelBundle or through the inline spec.models of a ModelDeployment. Only in an override can batch_sizes be set to the string "*", which selects every batch size the profile’s defaultBatchingConfig provides for that tier:

spec:
  modelConfigs:
  - model: gpt-oss-120b:1
    profile: gpt-oss-fp8-dyt
    batchingConfig:
      128k:
        batch_sizes: "*"    # every batch size the profile defines for 128k
      32k:
        batch_sizes: [2, 4] # a subset of what the profile defines for 32k

An override can only narrow or restate what the profile already provides. It cannot introduce a batch size that the profile’s defaultBatchingConfig does not contain.

PEFs

A PEF is the compiled artifact for one model configuration. The profile selects which PEF, sequence size, and batch size are used, so you reference a profile rather than a PEF. For the Pef resource reference, see Deploying custom checkpoints.

ModelBundle structure

A ModelBundle combines one or more model and profile pairs into a single named, deployable unit. Each entry in spec.modelConfigs references a model and a profile by name and can specify per-model batching and model settings. The following example shows a multi-model bundle:

apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
  name: mb-gpt120-llama70sd8-llama8
spec:
  modelConfigs:
    - model: gpt-oss-120b:1
      profile: gpt-oss-fp8-dyt
      batchingConfig:
        8k:
          batch_sizes: [2]
        32k:
          batch_sizes: [2]
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
      modelSettings:
        swappable: true
    - model: meta-llama-3-1-8b-instruct:1
      profile: llama-3p1-8b
      modelSettings:
        routable: false
  specDecodingPairs:
    - draft: meta-llama-3-1-8b-instruct
      target: meta-llama-3-3-70b-instruct

The example above includes three models, with Meta-Llama-3.1-8B-Instruct serving as the non-routable draft model for Meta-Llama-3.3-70B-Instruct. ModelBundle Top-Level Fields

Field	Required	Description
`spec.modelConfigs`	Yes	List of model configurations in the bundle, with one entry per model. See Model configurations.
`spec.specDecodingPairs`	No	Speculative decoding pairs. See the speculative decoding parameters below.
`spec.adapters`	No	LoRA adapter definitions used by the model configurations, keyed by adapter name.
`spec.skip_legalizer`	No	Skips legalizer validation for the bundle.

Model configurations

Each entry in spec.modelConfigs pairs one model with one profile:

Field	Required	Description
`model`	Yes	`Model` reference in `<model-name>[:<arch>][:<version>]` format. The architecture segment can be omitted when the model has a single architecture, and the version can be omitted to use the latest version.
`profile`	Conditional	Name of the `ModelProfile` this model uses. Set exactly one of `profile` or `profileDefinition`.
`profileDefinition`	Conditional	Inline profile specification, used when a model’s PEFs span multiple standard profiles. Set exactly one of `profile` or `profileDefinition`.
`batchingConfig`	No	Per-bundle override of the profile’s `defaultBatchingConfig`, using the same tier structure. A tier can be set to `"*"` to select every batch size the profile provides for it. See Overriding a batching configuration.
`modelSettings.swappable`	No	Whether the model is swappable. See Swappable models.
`modelSettings.routable`	No	Whether the model is directly routable by clients. Omit to inherit the default, which is routable. Set to `false` for speculative decoding draft models.
`modelSettings.checkpointOverrides`	No	Checkpoint selection applied in place of the `model` reference, using `checkpoint` for a model-level checkpoint and `expertOverrides` for per-expert checkpoints.
`modelSettings.adapters`	No	Mapping from serving name to a LoRA adapter binding, referencing an adapter defined in `spec.adapters`.
`modelSettings.properties`	No	Additional configuration fields applied across all of the model’s experts.

Batching is resolved in the following order of precedence: the model configuration’s batchingConfig, then the profile’s defaultBatchingConfig, then a configuration generated from the PEF grouping.

Swappable models

The swappable setting controls whether a model can be evicted from HBM in favor of another model on the same endpoint. When swappable is set to true, the entire model is removed from HBM if a different model needs to run on that endpoint. This includes the model’s checkpoint weights, its profile metadata, and its PEFs. Set this field to true for models that can be evicted in order to free capacity for another model, and leave it unset or set it to false for models that must remain resident.

Speculative decoding

In speculative decoding, a smaller draft model proposes tokens and a larger target model verifies them. The pairing is declared at the bundle level in spec.specDecodingPairs, and the draft model is included as its own non-routable entry in spec.modelConfigs. For the field reference, requirements, and the full workflow including custom draft checkpoints, see Deploying with speculative decoding.

ModelDeployment structure

A ModelDeployment instantiates models on the cluster and creates the serving pods behind a routable inference endpoint. For detailed deployment information, see Quickstart - Hosted or Quickstart - On-prem. Set exactly one model source:

Deployment path	Field	Description
Direct deployment	`spec.models`	Declares a model and profile inline. The specification is validated and legalized in place, and no bundle is used.
Bundle deployment	`spec.bundle`	References an existing `ModelBundle` by name. The bundle is trusted once it has been validated.

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120-llama70sd8-llama8
spec:
  bundle: mb-gpt120-llama70sd8-llama8
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200

Field	Required	Description
`spec.bundle`	Conditional	Name of the `ModelBundle` to deploy. Set exactly one of `spec.bundle` or `spec.models`.
`spec.models`	Conditional	Inline definition of the models and profiles to deploy, using the same `modelConfigs` structure as a `ModelBundle`. Set exactly one of `spec.bundle` or `spec.models`.
`spec.groups[].name`	Yes	Name identifier for the deployment group. Defaults to `default`.
`spec.groups[].minReplicas`	Yes	Minimum number of replicas to maintain.
`spec.groups[].qosList`	Yes	Quality of service classes for request prioritization.
`spec.owner`	Yes	Email address of the deployment owner for tracking and notifications.
`spec.secretNames`	Yes	Credentials used to access artifacts. Must match secrets configured in your environment.
`spec.engineConfig`	No	Configures the inference engine image, including settings such as `startupTimeout`.
`spec.cacheConfig`	No	Configures the artifact cache container.
`spec.tokenizerConfig`	No	Configures the tokenizer image.
`spec.storage`, `spec.storageClass`	No	Storage configuration for the deployment.
`spec.nodeSelector`, `spec.tolerations`	No	Default scheduling controls for the deployment.
`spec.skip_legalizer`	No	Skips the legalizer when deploying an inline `spec.models` specification.

Legalizer results

The legalizer validates a configuration’s resource requirements before deployment. It runs automatically when you apply a ModelBundle, and its results are published under status.legalizerInfo. The bundle status also reports the effective batching configuration that was resolved for each model.

Hosted
On Premise

kubectl describe modelbundle <bundle-name>

kubectl -n <namespace> describe modelbundle.sambanova.ai <bundle-name>

Example status for a bundle that passed with warnings:

Status:
  Batching Configs:
    gemma-4-31b-it:1:
      128k:
        batch_sizes:
          2
      32k:
        batch_sizes:
          2
          4
          6
          8
    minimax-m2-7:1:
      192k:
        batch_sizes:
          2
      8k:
        batch_sizes:
          2
          4
          6
          8
  Conditions:
    Last Transition Time:  2026-07-28T22:15:18.967900+00:00
    Message:               ModelBundle is Valid
    Observed Generation:   2
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  Valid
  Legalizer Info:
    Errors:
    Status:  Legalizer passed
    Utilization:
      Ddr:           0.0614
      hbm_resident:  0.0
      Host:          0.0304
    Warnings:
      gemma-4-31b-it-vision-ckpt: checkpoint sharing not enabled for gemma-4-31B-it-128k-vision-emb and gemma-4-31B-it-32k-vision-emb
  Observed Generation:  2

Field	Description
`status.conditions[].reason`	`ValidationSucceeded` or `ValidationFailed`. The accompanying `message` reads `ModelBundle is Valid` on success, or contains the error detail on failure.
`status.batchingConfigs`	Effective batching configuration per model, keyed by the model reference and then by sequence length tier. This reflects the configuration actually resolved from the model configuration, the profile’s default, or the PEF grouping.
`status.legalizerInfo.status`	Human-readable legalizer result: `Legalizer passed`, `Legalizer failed`, or `Legalizer was skipped`. Absent if legalizer output could not be processed.
`status.legalizerInfo.errors`	List of validation errors from the legalizer. Empty when the legalizer passed.
`status.legalizerInfo.warnings`	List of validation warnings, each in `<checkpoint-name>: <message>` form. Warnings do not block deployment.
`status.legalizerInfo.utilization.ddr`	DDR memory utilization as a decimal fraction (for example, `0.0614` represents approximately 6%)
`status.legalizerInfo.utilization.hbm_resident`	HBM resident memory utilization as a decimal fraction. Values above `1.0` indicate over-allocation.
`status.legalizerInfo.utilization.host`	Host memory utilization as a decimal fraction (for example, `0.0304` represents approximately 3%)

A bundle can pass the legalizer with warnings present, as in the example above, and still report ValidationSucceeded. Review warnings before deploying, but they do not prevent deployment.

The utilization fields show N/A when skip_legalizer: true is set on a bundle that has been legalized at least once. If the bundle has never been legalized, for example because skip_legalizer: true was set from initial deployment, legalizerInfo is absent entirely. The utilization field may also be absent if the legalizer output could not be parsed.

Procedures

Identify available models and profiles

Before authoring a deployment, identify the models and profiles available in your cluster and determine which pair together.

List available models

List the models and note the architecture keys under each model’s spec.checkpoints:

Hosted
On Premise

kubectl get models

Example:

kubectl describe model gpt-oss-120b

kubectl -n <namespace> get models.sambanova.ai

Example:

kubectl -n <namespace> describe model.sambanova.ai gpt-oss-120b

List available profiles

List the profiles and note the model_arch value of each:

Hosted
On Premise

kubectl get modelprofiles

kubectl -n <namespace> get modelprofiles.sambanova.ai

Output:

deepseek              17h
gpt-oss-fp8-dyt       17h
llama-3p1-70b-sd      17h
llama-3p1-8b          17h

Match a profile to a model

View a profile to confirm its architecture, features, and batching support:

Hosted
On Premise

kubectl describe modelprofile <profile-name>

kubectl -n <namespace> describe modelprofile.sambanova.ai <profile-name>

A profile can be used with a model when the profile’s model_arch value matches one of the architecture keys under the model’s spec.checkpoints. The operator does not verify this pairing, so confirm it yourself before deploying. Review the profile for:

model_arch - The architecture the profile targets
features - Optional features the profile’s PEFs support
defaultBatchingConfig - Sequence length tiers and their batch sizes
pefs - The PEF versions the profile references

A profile whose features list includes speculative_decoding requires a draft model. Pair it using spec.specDecodingPairs in a bundle.

Deploy a single model

To serve one model, pair it with a profile and deploy it directly. No bundle is created.

Create the YAML file

Create a YAML file that declares the model and profile inline under spec.models:

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120
spec:
  models:
    modelConfigs:
    - model: gpt-oss-120b:1
      profile: gpt-oss-fp8-dyt
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200

A checkpoint path is not required, because the checkpoint is resolved from the referenced Model.

Apply the ModelDeployment

Hosted
On Premise

kubectl apply -f <modeldeployment-file>.yaml

kubectl -n <namespace> apply -f <modeldeployment-file>.yaml

Monitor deployment status

Hosted
On Premise

kubectl get modeldeployments
kubectl describe modeldeployment <deployment-name>

kubectl -n <namespace> get modeldeployments.sambanova.ai
kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>

The deployment is complete once all of its serving pods report a ready status.

Create a ModelBundle

Create a bundle to serve several models as one unit, to configure speculative decoding, or to validate and share a named configuration.

Create the YAML file

Create a YAML file for your ModelBundle:

apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
  name: mb-gpt120
spec:
  modelConfigs:
  - model: gpt-oss-120b:1
    profile: gpt-oss-fp8-dyt
    batchingConfig:
      8k:
        batch_sizes: [2]
      32k:
        batch_sizes: [2]
      64k:
        batch_sizes: [2]
      128k:
        batch_sizes: [2]

For multi-model bundles with speculative decoding, see the ModelBundle Structure example.

Apply the ModelBundle

Hosted
On Premise

kubectl apply -f <modelbundle-file>.yaml

kubectl -n <namespace> apply -f <modelbundle-file>.yaml

Verify legalizer validation

The legalizer automatically runs when you apply the bundle and validates whether the configuration fits in RDU memory.

Hosted
On Premise

kubectl describe modelbundle <bundle-name>

kubectl -n <namespace> describe modelbundle.sambanova.ai <bundle-name>

Successful validation
Failed validation

Status:
  Conditions:
    Last Transition Time:  2026-06-22T21:11:05.689262+00:00
    Message:               ModelBundle is Valid
    Observed Generation:   1
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  Valid

Status:
  Conditions:
    Last Transition Time:  2026-06-22T21:11:54.975311+00:00
    Message:               <error-details>
    Reason:                ValidationFailed
    Status:                False
    Type:                  Valid

The Message field contains error details, including legalizer errors if any.

To read the legalizer result directly, including memory utilization and any warnings:

kubectl get modelbundle <bundle-name> -o jsonpath='{.status.legalizerInfo}'

For the full status structure and field descriptions, see Legalizer results.

Do not proceed to deployment until the bundle shows ValidationSucceeded.

Deploy a bundle

This procedure applies both to bundles that SambaNova provides and to bundles you authored.

Discover the available bundles

List the bundles present in the cluster and note the name of the one you want to deploy:

Hosted
On Premise

kubectl get modelbundles

kubectl -n <namespace> get modelbundles.sambanova.ai

Output:

mb-gpt120                      17h
mb-gpt120-llama70sd8-llama8    17h
ds-v32-gemma-4-31b             3h

Before deploying, confirm the bundle passed validation, as described in Legalizer results.

Create a ModelDeployment

Insert the bundle name into spec.bundle:

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120
spec:
  bundle: mb-gpt120
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader

Apply the ModelDeployment

Hosted
On Premise

kubectl apply -f <modeldeployment-file>.yaml

kubectl -n <namespace> apply -f <modeldeployment-file>.yaml

Monitor deployment status

Hosted
On Premise

kubectl get modeldeployments
kubectl describe modeldeployment <deployment-name>

kubectl -n <namespace> get modeldeployments.sambanova.ai
kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>

Switch bundles

A ModelDeployment’s bundle reference cannot be changed in place. To serve a different bundle, delete the existing deployment and apply a new one that references the new bundle.

Delete the existing ModelDeployment

Hosted
On Premise

kubectl delete modeldeployment <deployment-name>

kubectl -n <namespace> delete modeldeployment <deployment-name>

Apply a ModelDeployment for the new bundle

Author a new ModelDeployment whose spec.bundle names the bundle you want to serve, then apply it as described in Deploy a bundle.

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-qwen3-32b-whisper
spec:
  bundle: mb-qwen3-32b-whisper
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader

To request new bundles, contact SambaNova support.

Deploy multiple bundles

To serve more than one bundle at a time, apply a separate ModelDeployment for each bundle.

SambaStack supports only one bundle per node. When deploying multiple bundles, assign each bundle to separate nodes to avoid resource conflicts. Use spec.nodeSelector on each ModelDeployment to control placement.

Verify that the pods reflect the deployed bundles:

Hosted
On Premise

kubectl get pods

kubectl -n <namespace> get pods

Deploy a custom checkpoint

To serve your own converted or fine-tuned checkpoint, either override the checkpoint used by an existing model or register a new model for it. See Deploying custom checkpoints.

Add a custom model architecture

Serving a model architecture that SambaStack does not yet support requires compiling a PEF and authoring Pef, ModelProfile, and Model resources. See Deploying custom checkpoints.

Update or remove a deployment or bundle

Update a configuration
Remove a configuration

Modify the YAML file

Edit the ModelBundle, ModelProfile, or ModelDeployment YAML file with your changes.

Reapply the configuration

Hosted
On Premise

kubectl apply -f <modified-file>.yaml

kubectl -n <namespace> apply -f <modified-file>.yaml

The legalizer automatically revalidates the changes. Changing a profile or a model triggers a reconcile of every bundle and deployment that references it.

Delete the ModelDeployment

Hosted
On Premise

kubectl delete modeldeployment <deployment-name>

kubectl -n <namespace> delete modeldeployment <deployment-name>

Delete the ModelBundle (if one was used)

Hosted
On Premise

kubectl delete modelbundle <bundle-name>

kubectl -n <namespace> delete modelbundle <bundle-name>

Delete the ModelProfile (optional)

Delete a profile only if you authored it for a custom architecture. Profiles provided with the SambaStack installation are shared across models and should be left in place.

Hosted
On Premise

kubectl delete modelprofile <profile-name>

kubectl -n <namespace> delete modelprofile <profile-name>

Troubleshooting

Legalizer validation failures

Errors appear in status.legalizerInfo.errors and cause ValidationFailed. Warnings appear in status.legalizerInfo.warnings and do not block deployment. See Legalizer results.

Error Pattern	Cause	Resolution
`PEF pef1 and pef2 are not checkpoint compatible (checkpoint #0)`	PEFs with the same `ckpt_sharing_uuid` cannot share checkpoint memory	Assign different `ckpt_sharing_uuid` values to the incompatible PEFs
`<checkpoint>: checkpoint sharing not enabled for <expert-a> and <expert-b>`	Warning only. The two experts cannot share checkpoint memory, so each holds its own copy	No action is required. To reclaim the duplicated memory, reduce the number of tiers or use PEFs that can share a checkpoint
`Bundle exceeds memory constraints`	Combined PEF and checkpoint size exceeds RDU memory	Reduce the number of tiers or batch sizes in the batching configuration
`PEF not found: <pef-name>`	Referenced PEF does not exist	Verify the PEF name with `kubectl get pefs`
Profile or model reference cannot be resolved	The named `ModelProfile` or `Model` does not exist in the namespace	Verify names with `kubectl get modelprofiles` and `kubectl get models`

Deployment failures

Symptom	Possible Cause	Resolution
Deployment stuck in pending	Insufficient RDU resources	Check cluster capacity; reduce `minReplicas`
Checkpoint download fails	Invalid checkpoint path or missing credentials	Verify the `source` path in the Model; confirm the `sambanova-artifact-reader` secret exists
Model not accessible via API	Model name mismatch, or the model is not routable	Verify `spec.name` on the Model; confirm `modelSettings.routable` is not set to `false`
Draft model served directly	The draft model was left routable	Set `modelSettings.routable` to `false` on the draft model entry

Migrating bundle configuration

Migrate bundles built with the deprecated custom resources

Supported Models and Bundles

Catalogue of models and bundles available for deployment

Custom checkpoint deployment

Deploy your own custom or fine-tuned checkpoints

Checkpoint Conversion Tool

Convert Checkpoints to Compatible formats

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Prerequisites

Quickstart - Hosted

Quickstart - On-prem

Terminology

Concepts

Resource architecture

ModelProfile structure

Batching configuration

Overriding a batching configuration

PEFs

ModelBundle structure

Model configurations

Swappable models

Speculative decoding

ModelDeployment structure

Legalizer results

Procedures

Identify available models and profiles

Deploy a single model

Create a ModelBundle

Deploy a bundle

Switch bundles

Deploy multiple bundles

Deploy a custom checkpoint

Add a custom model architecture

Update or remove a deployment or bundle

Troubleshooting

Legalizer validation failures

Deployment failures

Migrating bundle configuration

Supported Models and Bundles

Custom checkpoint deployment

Checkpoint Conversion Tool

​Prerequisites

Quickstart - Hosted

Quickstart - On-prem

​Terminology

​Concepts

​Resource architecture

​ModelProfile structure

​Batching configuration

​Overriding a batching configuration

​PEFs

​ModelBundle structure

​Model configurations

​Swappable models

​Speculative decoding

​ModelDeployment structure

​Legalizer results

​Procedures

​Identify available models and profiles

​Deploy a single model

​Create a ModelBundle

​Deploy a bundle

​Switch bundles

​Deploy multiple bundles

​Deploy a custom checkpoint

​Add a custom model architecture

​Update or remove a deployment or bundle

​Troubleshooting

​Legalizer validation failures

​Deployment failures

​Related documentation

Migrating bundle configuration

Supported Models and Bundles

Custom checkpoint deployment

Checkpoint Conversion Tool

Prerequisites

Terminology

Concepts

Resource architecture

ModelProfile structure

Batching configuration

Overriding a batching configuration

PEFs

ModelBundle structure

Model configurations

Swappable models

Speculative decoding

ModelDeployment structure

Legalizer results

Procedures

Identify available models and profiles

Deploy a single model

Create a ModelBundle

Deploy a bundle

Switch bundles

Deploy multiple bundles

Deploy a custom checkpoint

Add a custom model architecture

Update or remove a deployment or bundle

Troubleshooting

Legalizer validation failures

Deployment failures

Related documentation