SambaStack ModelBundle Custom Resource - SambaNova Documentation

A ModelBundle combines one or more model and profile pairs into a single named, deployable unit. Each entry in spec.modelConfigs references a Model and a ModelProfile by name and can specify per-model batching and model settings. ModelBundle resources use apiVersion: sambanova.ai/v1alpha1.

A bundle is optional. To serve a single model, pair it with a profile and deploy it directly. Create a bundle when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration.

Structure

The following example shows a multi-model bundle:

apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
  name: mb-gpt120-llama70sd8-llama8
spec:
  modelConfigs:
    - model: gpt-oss-120b:1
      profile: gpt-oss-fp8-dyt
      batchingConfig:
        8k:
          batch_sizes: [2]
        32k:
          batch_sizes: [2]
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
      modelSettings:
        swappable: true
    - model: meta-llama-3-1-8b-instruct:1
      profile: llama-3p1-8b
      modelSettings:
        routable: false
  specDecodingPairs:
    - draft: meta-llama-3-1-8b-instruct
      target: meta-llama-3-3-70b-instruct

The example above includes three models, with Meta-Llama-3.1-8B-Instruct serving as the non-routable draft model for Meta-Llama-3.3-70B-Instruct. ModelBundle Top-Level Fields

Field	Required	Description
`spec.modelConfigs`	Yes	List of model configurations in the bundle, with one entry per model. See Model configurations.
`spec.specDecodingPairs`	No	Speculative decoding pairs. See the speculative decoding parameters below.
`spec.adapters`	No	LoRA adapter definitions used by the model configurations, keyed by adapter name.
`spec.skip_legalizer`	No	Skips legalizer validation for the bundle.

Model configurations

Each entry in spec.modelConfigs pairs one model with one profile:

Field	Required	Description
`model`	Yes	`Model` reference in `<model-name>[:<arch>][:<version>]` format. The architecture segment can be omitted when the model has a single architecture, and the version can be omitted to use the latest version.
`profile`	Conditional	Name of the `ModelProfile` this model uses. Set exactly one of `profile` or `profileDefinition`.
`profileDefinition`	Conditional	Inline profile specification, used when a model’s PEFs span multiple standard profiles. Set exactly one of `profile` or `profileDefinition`.
`batchingConfig`	No	Per-bundle override of the profile’s batching. Either a string naming one of the profile’s `batchingConfigs` entries, such as `all` or `recommended`, or an inline tier map in which a tier can be set to `"*"` to select every batch size the profile provides for it. See Overriding a batching configuration.
`modelSettings.swappable`	No	Whether the model is swappable. See Swappable models.
`modelSettings.routable`	No	Whether the model is directly routable by clients. Omit to inherit the default, which is routable. Set to `false` for speculative decoding draft models.
`modelSettings.checkpointOverrides`	No	Checkpoint selection applied in place of the `model` reference, using `checkpoint` for a model-level checkpoint and `expertOverrides` for per-expert checkpoints.
`modelSettings.adapters`	No	Mapping from serving name to a LoRA adapter binding, referencing an adapter defined in `spec.adapters`.
`modelSettings.properties`	No	Additional configuration fields applied across all of the model’s experts.

Batching is resolved in the following order of precedence: the model configuration’s batchingConfig, then the profile’s recommended configuration, then its all configuration, then a configuration generated from the PEF grouping. See Resolution order.

Swappable models

The swappable setting controls whether a model can be evicted from HBM in favor of another model on the same endpoint. When swappable is set to true, the entire model is removed from HBM if a different model needs to run on that endpoint. This includes the model’s checkpoint weights, its profile metadata, and its PEFs. Set this field to true for models that can be evicted in order to free capacity for another model, and leave it unset or set it to false for models that must remain resident.

Speculative decoding

In speculative decoding, a smaller draft model proposes tokens and a larger target model verifies them. The pairing is declared at the bundle level in spec.specDecodingPairs, and the draft model is included as its own non-routable entry in spec.modelConfigs. For the field reference, requirements, and the full workflow including custom draft checkpoints, see Deploying with speculative decoding.

Legalizer results

The legalizer validates a configuration’s resource requirements before deployment. It runs automatically when you apply a ModelBundle, and its results are published under status.legalizerInfo. The bundle status also reports the effective batching configuration that was resolved for each model.

Hosted
On Premise

kubectl describe modelbundle <bundle-name>

kubectl -n <namespace> describe modelbundle.sambanova.ai <bundle-name>

Example status for a bundle that passed with warnings:

Status:
  Batching Configs:
    gemma-4-31b-it:1:
      128k:
        batch_sizes:
          2
      32k:
        batch_sizes:
          2
          4
          6
          8
    minimax-m2-7:1:
      192k:
        batch_sizes:
          2
      8k:
        batch_sizes:
          2
          4
          6
          8
  Conditions:
    Last Transition Time:  2026-07-28T22:15:18.967900+00:00
    Message:               ModelBundle is Valid
    Observed Generation:   2
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  Valid
  Legalizer Info:
    Errors:
    Status:  Legalizer passed
    Utilization:
      Ddr:           0.0614
      hbm_resident:  0.0
      Host:          0.0304
    Warnings:
      gemma-4-31b-it-vision-ckpt: checkpoint sharing not enabled for gemma-4-31B-it-128k-vision-emb and gemma-4-31B-it-32k-vision-emb
  Observed Generation:  2

Field	Description
`status.conditions[].reason`	`ValidationSucceeded` or `ValidationFailed`. The accompanying `message` reads `ModelBundle is Valid` on success, or contains the error detail on failure.
`status.batchingConfigs`	Effective batching configuration per model, keyed by the model reference and then by sequence length tier. This reflects the configuration actually resolved from the model configuration, the profile’s default, or the PEF grouping.
`status.legalizerInfo.status`	Human-readable legalizer result: `Legalizer passed`, `Legalizer failed`, or `Legalizer was skipped`. Absent if legalizer output could not be processed.
`status.legalizerInfo.errors`	List of validation errors from the legalizer. Empty when the legalizer passed.
`status.legalizerInfo.warnings`	List of validation warnings, each in `<checkpoint-name>: <message>` form. Warnings do not block deployment.
`status.legalizerInfo.utilization.ddr`	DDR memory utilization as a decimal fraction (for example, `0.0614` represents approximately 6%)
`status.legalizerInfo.utilization.hbm_resident`	HBM resident memory utilization as a decimal fraction. Values above `1.0` indicate over-allocation.
`status.legalizerInfo.utilization.host`	Host memory utilization as a decimal fraction (for example, `0.0304` represents approximately 3%)

A bundle can pass the legalizer with warnings present, as in the example above, and still report ValidationSucceeded. Review warnings before deploying, but they do not prevent deployment.

The utilization fields show N/A when skip_legalizer: true is set on a bundle that has been legalized at least once. If the bundle has never been legalized, for example because skip_legalizer: true was set from initial deployment, legalizerInfo is absent entirely. The utilization field may also be absent if the legalizer output could not be parsed.

Create a custom bundle

Author, apply, and validate a bundle of your own

ModelDeployment

Serves a bundle on the cluster

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

ModelBundle

Structure

Model configurations

Swappable models

Speculative decoding

Legalizer results

Create a custom bundle

ModelDeployment

​Structure

​Model configurations

​Swappable models

​Speculative decoding

​Legalizer results

​Related resources

Create a custom bundle

ModelDeployment

Structure

Model configurations

Swappable models

Speculative decoding

Legalizer results

Related resources