> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploying models and bundles

> Deploy models and bundles using ModelProfile, ModelBundle, and ModelDeployment resources on SambaStack, including custom bundles with custom models and checkpoints.

This guide describes how to deploy models with specific configurations on SambaStack by authoring custom resources: a **ModelDeployment** to serve a single model, or a **ModelBundle** to serve several models together.

In SambaStack, a deployment serves one or more models on a single node, together with their configurations, including batch sizes and sequence lengths. This approach uses the SambaNova Reconfigurable Dataflow Unit (RDU) to support multiple models and configurations in a single deployment. Because every model in a deployment is already resident, switching between them does not require reloading weights, which increases throughput compared with deployments that load a single static model.

For example, a single bundle can serve both `Llama-3.3-70B-Instruct` and `Llama-3.1-8B-Instruct`, allowing you to switch between them almost instantly. Each configuration occupies space on the node, so different bundles contain different sets of configurations. A copy of `Llama-3.3-70B-Instruct` with a batch size of 4 and a sequence length of 16k represents one configuration. For the bundles that SambaNova provides, see [Supported Models and Bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/supported-models-and-bundles).

<Note>
  SambaStack v2.0.2 introduces a new set of Kubernetes custom resources (CRs) to manage how bundles are deployed. The CRs in older versions of SambaStack are deprecated and will be functional until September 30, 2026. Model bundles built using the deprecated CRs will need to be [migrated](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/bundles-v3).
</Note>

<Note>
  A bundle is optional. To serve a single model, pair it with a model profile and deploy it directly. Create a bundle when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration.
</Note>

<Note>
  This guide covers both deploying the bundles that SambaNova provides and authoring your own. For the catalogue of provided bundles, see [Supported Models and Bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/supported-models-and-bundles).
</Note>

## Prerequisites

Before creating custom deployments or bundles, complete the following that applies to you:

<CardGroup cols={3}>
  <Card icon="list-check" href="/docs/en/sambastack/getting-started/hosted" title="Quickstart - Hosted">
    System set up for hosted SambaStack
  </Card>

  <Card icon="list-check" href="/docs/en/sambastack/getting-started/on-prem" title="Quickstart - On-prem">
    System set up for On-prem Sambastack
  </Card>
</CardGroup>

Additionally, review the following documentation:

* [Supported Models and Bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/supported-models-and-bundles) - Available model checkpoints
* [Speculative Decoding Deployment Guidelines](/docs/en/v2.0.2/sambastack/service-administration/performance/deploy-with-speculative-decoding) - Required if configuring speculative decoding pairs

### Terminology

| Term                     | Definition                                                                                                                                                                   |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **RDU**                  | Reconfigurable Dataflow Unit - SambaNova's proprietary processor architecture                                                                                                |
| **PEF**                  | Processor Executable Format - Compiled model binaries that run on RDUs                                                                                                       |
| **Bundle**               | A deployable package combining models, their profiles, and their configurations                                                                                              |
| **Configuration**        | The runtime settings for a model, including batch size and sequence length. A single deployment can include multiple configurations, enabling instant switching between them |
| **Model architecture**   | The `model_arch` value shared by a set of PEFs, which determines the checkpoints those PEFs are compatible with                                                              |
| **Expert**               | A sequence length profile configuration (for example, 8k, 16k, 32k) within a model                                                                                           |
| **Tier**                 | A sequence length key in a batching configuration (for example, `8k` or `128k`), which maps to the batch sizes served at that length                                         |
| **Speculative Decoding** | An optimization technique using a smaller draft model to accelerate inference from a larger target model                                                                     |
| **Legalizer**            | A validation process that verifies a configuration fits within RDU memory constraints                                                                                        |
| **CR (Custom Resource)** | A Kubernetes extension that defines custom resource types such as `Pef`, `Model`, `ModelProfile`, `ModelDeployment`, and `ModelBundle`                                       |

## Concepts

### Resource architecture

This section covers the resource structure used when creating **custom** deployments. All of these resources use `apiVersion: sambanova.ai/v1alpha1`.

```mermaid theme={}
graph TD
    PEF["PEF<br/><small>references to versioned executables</small>"]
    MP["ModelProfile<br/><small>set&nbsp;of&nbsp;feature‑compatible&nbsp;PEFs&nbsp;for&nbsp;a&nbsp;model&nbsp;arch<br/>+&nbsp;default&nbsp;batching&nbsp;configuration</small>"]
    M["Model<br/><small>checkpoints per architecture</small>"]
    MB["ModelBundle<br/><small>reusable,&nbsp;shareable&nbsp;set&nbsp;of<br/>models&nbsp;+&nbsp;profiles</small>"]
    MD["ModelDeployment<br/><small>routable inference endpoint, replicas &amp; QoS, backed by serving pods</small>"]
    JP("<i>ModelProfile & Model Pair</i><br/><small>implicit pairing, not a CR<br/>model_arch must be compatible</small>")

    PEF -->|"referenced by"| MP
    MP --- JP
    M --- JP
    JP -->|"referenced by"| MB
    MB ==>|"bundle deploy"| MD
    JP -.->|"direct deploy"| MD

    classDef cluster fill:#eef2f7,stroke:#8aa0bd,color:#1a2b45;
    classDef authored fill:#cfe2f3,stroke:#2f6fb0,color:#0d2c4d,stroke-width:2px;
    classDef config fill:#fbe7c6,stroke:#c9871f,color:#5c3d00;
    class PEF,MP,M cluster;
    class MB,MD authored;
    class JP config;
```

Light boxes are provided with the SambaStack installation and referenced by name; darker boxes are what you author. The amber box is not a custom resource, but a single `Model` and `ModelProfile` pairing, which corresponds to one entry in `spec.modelConfigs`.

The resources divide responsibility as follows:

1. **Pef** - Registers a compiled executable and the metadata describing what it supports, including its model architecture, batch size, and maximum sequence length.
2. **ModelProfile** - Defines *how* a model architecture can be run: which PEFs it uses, which features it supports, and the batch sizes available at each sequence length tier.
3. **Model** - Defines *which checkpoints* are available for each architecture, along with the tokenizer to use.
4. **ModelBundle** - Combines one or more model and profile pairs into a single named, deployable unit. Optional.
5. **ModelDeployment** - Instantiates one or more replicas on the cluster, either from a bundle or from an inline model and profile pair.

This separation allows you to:

* Deploy a single model by pairing it with a profile, without creating a bundle
* Reuse one profile across every model and checkpoint that shares its architecture, including custom checkpoints for fine-tuned models
* Serve a custom checkpoint by adding a `Model` resource only, leaving the profile untouched
* Group several models into one validated, shareable unit when you need them served together
* Update checkpoints without modifying profiles or deployment configurations
* Deploy the same configuration with different replica counts

<Note>
  Checkpoint paths are never declared in a bundle or a deployment. The operator resolves them from the `Model` you reference, either through the `<model-name>[:<arch>][:<version>]` reference or through `modelSettings.checkpointOverrides`.

  Compatibility between a checkpoint and a profile is defined by `model_arch`, but the operator does not verify it. Select a checkpoint whose architecture is compatible with the profile you reference.
</Note>

### ModelProfile structure

A ModelProfile defines the runtime shape of a single model architecture. Profiles are reusable, and the profiles for supported architectures are provided with the SambaStack installation, so in most cases you reference an existing profile by name rather than authoring one.

```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: ModelProfile
metadata:
  name: gpt-oss-fp8-dyt
spec:
  model_arch: gpt-oss-fp8
  features: []
  defaultBatchingConfig:
    8k:
      batch_sizes: [2, 4, 6, 8]
    32k:
      batch_sizes: [2, 4, 6, 8]
    64k:
      batch_sizes: [2, 4]
    128k:
      batch_sizes: [2]
  pefs:
    - gpt-oss-fp8-ss131072-bs8-dyt-1:1
  secretNames:
    - sambanova-artifact-reader
```

**ModelProfile Top-Level Fields**

| Field                        | Required | Description                                                                                                                                                                                                                               |
| ---------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec.pefs`                  | Yes      | Ordered list of PEF custom resource references, each in `<pef-name>[:<version>]` format. The list must be non-empty and must not contain duplicate PEF names. If the version suffix is omitted, the latest version is used.               |
| `spec.model_arch`            | Yes      | Model architecture targeted by this profile. All PEFs in a profile share the same `model_arch`, and this field makes that value visible without inspecting the individual `Pef` resources.                                                |
| `spec.features`              | Yes      | Features supported by this profile, derived from its PEFs. Known values are `continuous_batching`, `prompt_caching`, `speculative_decoding`, `speculative_prefill`, and `constrained_decoding`. An empty list means no optional features. |
| `spec.defaultBatchingConfig` | Yes      | Default batching support, keyed by sequence length tier. See [Batching configuration](#batching-configuration). If omitted, the operator generates the configuration from the PEF grouping.                                               |
| `spec.secretNames`           | Yes      | List of Kubernetes secrets required in order to download the PEFs. Must match secrets configured in your environment.                                                                                                                     |
| `spec.skip_legalizer`        | No       | Skips legalizer validation for this profile.                                                                                                                                                                                              |
| `status.batchingConfig`      | N/A      | Read-only. The resolved default batching configuration, taken from the spec or generated by the operator.                                                                                                                                 |

#### Batching configuration

A batching configuration maps each sequence length tier to the batch sizes served at that length. Common tier keys are `8k`, `16k`, `32k`, `64k`, and `128k`. PEFs below 1k use a bare integer key, for example `448`, and vocoder PEFs use a codes-length key in the form `<n>t`, for example `10t`.

```yaml theme={}
spec:
  defaultBatchingConfig:
    128k:
      batch_sizes: [1]
    32k:
      batch_sizes: [1, 2, 4]
    8k:
      batch_sizes: [1, 2, 4, 8]
    4k:
      batch_sizes: [1, 2, 4, 8]
      is_default: true
```

| Parameter     | Required | Description                                                                                                                                                                                                              |
| ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `batch_sizes` | Yes      | In a `defaultBatchingConfig`, an explicit list of integers, for example `[1, 2, 4]`. In an override, this can also be the string `"*"`. See [Overriding a batching configuration](#overriding-a-batching-configuration). |
| `is_default`  | No       | If `true`, this tier is served as the `default` expert. At most one tier per batching configuration may set this.                                                                                                        |

<Tip>
  Including multiple batch sizes for each tier allows the inference engine to select the smallest and fastest configuration based on current workload.
</Tip>

#### Overriding a batching configuration

A `ModelProfile` establishes the batch sizes available at each tier, so a `defaultBatchingConfig` always lists them explicitly. It never uses `"*"`.

The tier-to-`batch_sizes` structure is reused when you override a profile's defaults for one model, either through `modelConfigs[].batchingConfig` in a ModelBundle or through the inline `spec.models` of a ModelDeployment. Only in an override can `batch_sizes` be set to the string `"*"`, which selects every batch size the profile's `defaultBatchingConfig` provides for that tier:

```yaml theme={}
spec:
  modelConfigs:
  - model: gpt-oss-120b:1
    profile: gpt-oss-fp8-dyt
    batchingConfig:
      128k:
        batch_sizes: "*"    # every batch size the profile defines for 128k
      32k:
        batch_sizes: [2, 4] # a subset of what the profile defines for 32k
```

An override can only narrow or restate what the profile already provides. It cannot introduce a batch size that the profile's `defaultBatchingConfig` does not contain.

#### PEFs

A PEF is the compiled artifact for one model configuration. The profile selects which PEF, sequence size, and batch size are used, so you reference a profile rather than a PEF. For the `Pef` resource reference, see [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints#pef-structure).

### ModelBundle structure

A ModelBundle combines one or more model and profile pairs into a single named, deployable unit. Each entry in `spec.modelConfigs` references a model and a profile by name and can specify per-model batching and model settings.

The following example shows a multi-model bundle:

```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
  name: mb-gpt120-llama70sd8-llama8
spec:
  modelConfigs:
    - model: gpt-oss-120b:1
      profile: gpt-oss-fp8-dyt
      batchingConfig:
        8k:
          batch_sizes: [2]
        32k:
          batch_sizes: [2]
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
      modelSettings:
        swappable: true
    - model: meta-llama-3-1-8b-instruct:1
      profile: llama-3p1-8b
      modelSettings:
        routable: false
  specDecodingPairs:
    - draft: meta-llama-3-1-8b-instruct
      target: meta-llama-3-3-70b-instruct
```

The example above includes three models, with Meta-Llama-3.1-8B-Instruct serving as the non-routable draft model for Meta-Llama-3.3-70B-Instruct.

**ModelBundle Top-Level Fields**

| Field                    | Required | Description                                                                                                              |
| ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `spec.modelConfigs`      | Yes      | List of model configurations in the bundle, with one entry per model. See [Model configurations](#model-configurations). |
| `spec.specDecodingPairs` | No       | Speculative decoding pairs. See the speculative decoding parameters below.                                               |
| `spec.adapters`          | No       | LoRA adapter definitions used by the model configurations, keyed by adapter name.                                        |
| `spec.skip_legalizer`    | No       | Skips legalizer validation for the bundle.                                                                               |

#### Model configurations

Each entry in `spec.modelConfigs` pairs one model with one profile:

| Field                               | Required    | Description                                                                                                                                                                                                                                                      |
| ----------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`                             | Yes         | `Model` reference in `<model-name>[:<arch>][:<version>]` format. The architecture segment can be omitted when the model has a single architecture, and the version can be omitted to use the latest version.                                                     |
| `profile`                           | Conditional | Name of the `ModelProfile` this model uses. Set exactly one of `profile` or `profileDefinition`.                                                                                                                                                                 |
| `profileDefinition`                 | Conditional | Inline profile specification, used when a model's PEFs span multiple standard profiles. Set exactly one of `profile` or `profileDefinition`.                                                                                                                     |
| `batchingConfig`                    | No          | Per-bundle override of the profile's `defaultBatchingConfig`, using the same tier structure. A tier can be set to `"*"` to select every batch size the profile provides for it. See [Overriding a batching configuration](#overriding-a-batching-configuration). |
| `modelSettings.swappable`           | No          | Whether the model is swappable. See [Swappable models](#swappable-models).                                                                                                                                                                                       |
| `modelSettings.routable`            | No          | Whether the model is directly routable by clients. Omit to inherit the default, which is routable. **Set to `false` for speculative decoding draft models.**                                                                                                     |
| `modelSettings.checkpointOverrides` | No          | Checkpoint selection applied in place of the `model` reference, using `checkpoint` for a model-level checkpoint and `expertOverrides` for per-expert checkpoints.                                                                                                |
| `modelSettings.adapters`            | No          | Mapping from serving name to a LoRA adapter binding, referencing an adapter defined in `spec.adapters`.                                                                                                                                                          |
| `modelSettings.properties`          | No          | Additional configuration fields applied across all of the model's experts.                                                                                                                                                                                       |

Batching is resolved in the following order of precedence: the model configuration's `batchingConfig`, then the profile's `defaultBatchingConfig`, then a configuration generated from the PEF grouping.

#### Swappable models

The `swappable` setting controls whether a model can be evicted from HBM in favor of another model on the same endpoint. When `swappable` is set to `true`, the entire model is removed from HBM if a different model needs to run on that endpoint. This includes the model's checkpoint weights, its profile metadata, and its PEFs.

Set this field to `true` for models that can be evicted in order to free capacity for another model, and leave it unset or set it to `false` for models that must remain resident.

#### Speculative decoding

In speculative decoding, a smaller draft model proposes tokens and a larger target model verifies them. The pairing is declared at the bundle level in `spec.specDecodingPairs`, and the draft model is included as its own non-routable entry in `spec.modelConfigs`.

For the field reference, requirements, and the full workflow including custom draft checkpoints, see [Deploying with speculative decoding](/docs/en/v2.0.2/sambastack/service-administration/performance/deploy-with-speculative-decoding).

### ModelDeployment structure

A ModelDeployment instantiates models on the cluster and creates the serving pods behind a routable inference endpoint. For detailed deployment information, see [Quickstart - Hosted](/docs/en/v2.0.2/sambastack/getting-started/hosted) or [Quickstart - On-prem](/docs/en/v2.0.2/sambastack/getting-started/on-prem).

Set **exactly one** model source:

| Deployment path   | Field         | Description                                                                                                        |
| ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------ |
| Direct deployment | `spec.models` | Declares a model and profile inline. The specification is validated and legalized in place, and no bundle is used. |
| Bundle deployment | `spec.bundle` | References an existing `ModelBundle` by name. The bundle is trusted once it has been validated.                    |

```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120-llama70sd8-llama8
spec:
  bundle: mb-gpt120-llama70sd8-llama8
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200
```

| Field                                   | Required    | Description                                                                                                                                                            |
| --------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec.bundle`                           | Conditional | Name of the `ModelBundle` to deploy. Set exactly one of `spec.bundle` or `spec.models`.                                                                                |
| `spec.models`                           | Conditional | Inline definition of the models and profiles to deploy, using the same `modelConfigs` structure as a `ModelBundle`. Set exactly one of `spec.bundle` or `spec.models`. |
| `spec.groups[].name`                    | Yes         | Name identifier for the deployment group. Defaults to `default`.                                                                                                       |
| `spec.groups[].minReplicas`             | Yes         | Minimum number of replicas to maintain.                                                                                                                                |
| `spec.groups[].qosList`                 | Yes         | Quality of service classes for request prioritization.                                                                                                                 |
| `spec.owner`                            | Yes         | Email address of the deployment owner for tracking and notifications.                                                                                                  |
| `spec.secretNames`                      | Yes         | Credentials used to access artifacts. Must match secrets configured in your environment.                                                                               |
| `spec.engineConfig`                     | No          | Configures the inference engine image, including settings such as `startupTimeout`.                                                                                    |
| `spec.cacheConfig`                      | No          | Configures the artifact cache container.                                                                                                                               |
| `spec.tokenizerConfig`                  | No          | Configures the tokenizer image.                                                                                                                                        |
| `spec.storage`, `spec.storageClass`     | No          | Storage configuration for the deployment.                                                                                                                              |
| `spec.nodeSelector`, `spec.tolerations` | No          | Default scheduling controls for the deployment.                                                                                                                        |
| `spec.skip_legalizer`                   | No          | Skips the legalizer when deploying an inline `spec.models` specification.                                                                                              |

### Legalizer results

The legalizer validates a configuration's resource requirements before deployment. It runs automatically when you apply a ModelBundle, and its results are published under `status.legalizerInfo`. The bundle status also reports the effective batching configuration that was resolved for each model.

<Tabs>
  <Tab title="Hosted">
    ```bash theme={}
    kubectl describe modelbundle <bundle-name>
    ```
  </Tab>

  <Tab title="On Premise">
    ```bash theme={}
    kubectl -n <namespace> describe modelbundle.sambanova.ai <bundle-name>
    ```
  </Tab>
</Tabs>

Example status for a bundle that passed with warnings:

```text theme={}
Status:
  Batching Configs:
    gemma-4-31b-it:1:
      128k:
        batch_sizes:
          2
      32k:
        batch_sizes:
          2
          4
          6
          8
    minimax-m2-7:1:
      192k:
        batch_sizes:
          2
      8k:
        batch_sizes:
          2
          4
          6
          8
  Conditions:
    Last Transition Time:  2026-07-28T22:15:18.967900+00:00
    Message:               ModelBundle is Valid
    Observed Generation:   2
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  Valid
  Legalizer Info:
    Errors:
    Status:  Legalizer passed
    Utilization:
      Ddr:           0.0614
      hbm_resident:  0.0
      Host:          0.0304
    Warnings:
      gemma-4-31b-it-vision-ckpt: checkpoint sharing not enabled for gemma-4-31B-it-128k-vision-emb and gemma-4-31B-it-32k-vision-emb
  Observed Generation:  2
```

| Field                                           | Description                                                                                                                                                                                                                            |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status.conditions[].reason`                    | `ValidationSucceeded` or `ValidationFailed`. The accompanying `message` reads `ModelBundle is Valid` on success, or contains the error detail on failure.                                                                              |
| `status.batchingConfigs`                        | Effective batching configuration per model, keyed by the model reference and then by sequence length tier. This reflects the configuration actually resolved from the model configuration, the profile's default, or the PEF grouping. |
| `status.legalizerInfo.status`                   | Human-readable legalizer result: `Legalizer passed`, `Legalizer failed`, or `Legalizer was skipped`. Absent if legalizer output could not be processed.                                                                                |
| `status.legalizerInfo.errors`                   | List of validation errors from the legalizer. Empty when the legalizer passed.                                                                                                                                                         |
| `status.legalizerInfo.warnings`                 | List of validation warnings, each in `<checkpoint-name>: <message>` form. Warnings do not block deployment.                                                                                                                            |
| `status.legalizerInfo.utilization.ddr`          | DDR memory utilization as a decimal fraction (for example, `0.0614` represents approximately 6%)                                                                                                                                       |
| `status.legalizerInfo.utilization.hbm_resident` | HBM resident memory utilization as a decimal fraction. Values above `1.0` indicate over-allocation.                                                                                                                                    |
| `status.legalizerInfo.utilization.host`         | Host memory utilization as a decimal fraction (for example, `0.0304` represents approximately 3%)                                                                                                                                      |

<Note>
  A bundle can pass the legalizer with warnings present, as in the example above, and still report `ValidationSucceeded`. Review warnings before deploying, but they do not prevent deployment.
</Note>

<Note>
  The `utilization` fields show `N/A` when `skip_legalizer: true` is set on a bundle that has been legalized at least once. If the bundle has never been legalized, for example because `skip_legalizer: true` was set from initial deployment, `legalizerInfo` is absent entirely. The `utilization` field may also be absent if the legalizer output could not be parsed.
</Note>

## Procedures

### Identify available models and profiles

Before authoring a deployment, identify the models and profiles available in your cluster and determine which pair together.

<Steps>
  <Step title="List available models">
    List the models and note the architecture keys under each model's `spec.checkpoints`:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get models
        ```

        Example:

        ```bash theme={}
        kubectl describe model gpt-oss-120b
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get models.sambanova.ai
        ```

        Example:

        ```bash theme={}
        kubectl -n <namespace> describe model.sambanova.ai gpt-oss-120b
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="List available profiles">
    List the profiles and note the `model_arch` value of each:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get modelprofiles
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get modelprofiles.sambanova.ai
        ```
      </Tab>
    </Tabs>

    Output:

    ```text theme={}
    deepseek              17h
    gpt-oss-fp8-dyt       17h
    llama-3p1-70b-sd      17h
    llama-3p1-8b          17h
    ```
  </Step>

  <Step title="Match a profile to a model">
    View a profile to confirm its architecture, features, and batching support:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl describe modelprofile <profile-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> describe modelprofile.sambanova.ai <profile-name>
        ```
      </Tab>
    </Tabs>

    A profile can be used with a model when the profile's `model_arch` value matches one of the architecture keys under the model's `spec.checkpoints`. The operator does not verify this pairing, so confirm it yourself before deploying. Review the profile for:

    * `model_arch` - The architecture the profile targets
    * `features` - Optional features the profile's PEFs support
    * `defaultBatchingConfig` - Sequence length tiers and their batch sizes
    * `pefs` - The PEF versions the profile references
  </Step>
</Steps>

<Tip>
  A profile whose `features` list includes `speculative_decoding` requires a draft model. Pair it using `spec.specDecodingPairs` in a bundle.
</Tip>

### Deploy a single model

To serve one model, pair it with a profile and deploy it directly. No bundle is created.

<Steps>
  <Step title="Create the YAML file">
    Create a YAML file that declares the model and profile inline under `spec.models`:

    ```yaml theme={}
    apiVersion: sambanova.ai/v1alpha1
    kind: ModelDeployment
    metadata:
      name: md-gpt120
    spec:
      models:
        modelConfigs:
        - model: gpt-oss-120b:1
          profile: gpt-oss-fp8-dyt
      groups:
      - minReplicas: 1
        name: default
        qosList:
        - free
      owner: no-reply@sambanova.ai
      secretNames:
      - sambanova-artifact-reader
      engineConfig:
        startupTimeout: 7200
    ```

    A checkpoint path is not required, because the checkpoint is resolved from the referenced `Model`.
  </Step>

  <Step title="Apply the ModelDeployment">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl apply -f <modeldeployment-file>.yaml
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> apply -f <modeldeployment-file>.yaml
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Monitor deployment status">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get modeldeployments
        kubectl describe modeldeployment <deployment-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get modeldeployments.sambanova.ai
        kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>
        ```
      </Tab>
    </Tabs>

    The deployment is complete once all of its serving pods report a ready status.
  </Step>
</Steps>

### Create a ModelBundle

Create a bundle to serve several models as one unit, to configure speculative decoding, or to validate and share a named configuration.

<Steps>
  <Step title="Create the YAML file">
    Create a YAML file for your ModelBundle:

    ```yaml theme={}
    apiVersion: sambanova.ai/v1alpha1
    kind: ModelBundle
    metadata:
      name: mb-gpt120
    spec:
      modelConfigs:
      - model: gpt-oss-120b:1
        profile: gpt-oss-fp8-dyt
        batchingConfig:
          8k:
            batch_sizes: [2]
          32k:
            batch_sizes: [2]
          64k:
            batch_sizes: [2]
          128k:
            batch_sizes: [2]
    ```

    For multi-model bundles with speculative decoding, see the [ModelBundle Structure](#modelbundle-structure) example.
  </Step>

  <Step title="Apply the ModelBundle">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl apply -f <modelbundle-file>.yaml
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> apply -f <modelbundle-file>.yaml
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Verify legalizer validation">
    The legalizer automatically runs when you apply the bundle and validates whether the configuration fits in RDU memory.

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl describe modelbundle <bundle-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> describe modelbundle.sambanova.ai <bundle-name>
        ```
      </Tab>
    </Tabs>

    <Tabs>
      <Tab title="Successful validation">
        ```text theme={}
        Status:
          Conditions:
            Last Transition Time:  2026-06-22T21:11:05.689262+00:00
            Message:               ModelBundle is Valid
            Observed Generation:   1
            Reason:                ValidationSucceeded
            Status:                True
            Type:                  Valid
        ```
      </Tab>

      <Tab title="Failed validation">
        ```text theme={}
        Status:
          Conditions:
            Last Transition Time:  2026-06-22T21:11:54.975311+00:00
            Message:               <error-details>
            Reason:                ValidationFailed
            Status:                False
            Type:                  Valid
        ```

        The `Message` field contains error details, including legalizer errors if any.
      </Tab>
    </Tabs>

    To read the legalizer result directly, including memory utilization and any warnings:

    ```bash theme={}
    kubectl get modelbundle <bundle-name> -o jsonpath='{.status.legalizerInfo}'
    ```

    For the full status structure and field descriptions, see [Legalizer results](#legalizer-results).
  </Step>
</Steps>

<Warning>
  Do not proceed to deployment until the bundle shows `ValidationSucceeded`.
</Warning>

### Deploy a bundle

This procedure applies both to bundles that SambaNova provides and to bundles you authored.

<Steps>
  <Step title="Discover the available bundles">
    List the bundles present in the cluster and note the name of the one you want to deploy:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get modelbundles
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get modelbundles.sambanova.ai
        ```
      </Tab>
    </Tabs>

    Output:

    ```text theme={}
    mb-gpt120                      17h
    mb-gpt120-llama70sd8-llama8    17h
    ds-v32-gemma-4-31b             3h
    ```

    Before deploying, confirm the bundle passed validation, as described in [Legalizer results](#legalizer-results).
  </Step>

  <Step title="Create a ModelDeployment">
    Insert the bundle name into `spec.bundle`:

    ```yaml theme={}
    apiVersion: sambanova.ai/v1alpha1
    kind: ModelDeployment
    metadata:
      name: md-gpt120
    spec:
      bundle: mb-gpt120
      groups:
      - minReplicas: 1
        name: default
        qosList:
        - free
      owner: no-reply@sambanova.ai
      secretNames:
      - sambanova-artifact-reader
    ```
  </Step>

  <Step title="Apply the ModelDeployment">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl apply -f <modeldeployment-file>.yaml
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> apply -f <modeldeployment-file>.yaml
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Monitor deployment status">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get modeldeployments
        kubectl describe modeldeployment <deployment-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get modeldeployments.sambanova.ai
        kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

### Switch bundles

A ModelDeployment's bundle reference cannot be changed in place. To serve a different bundle, delete the existing deployment and apply a new one that references the new bundle.

<Steps>
  <Step title="Delete the existing ModelDeployment">
    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl delete modeldeployment <deployment-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> delete modeldeployment <deployment-name>
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Apply a ModelDeployment for the new bundle">
    Author a new ModelDeployment whose `spec.bundle` names the bundle you want to serve, then apply it as described in [Deploy a bundle](#deploy-a-bundle).

    ```yaml theme={}
    apiVersion: sambanova.ai/v1alpha1
    kind: ModelDeployment
    metadata:
      name: md-qwen3-32b-whisper
    spec:
      bundle: mb-qwen3-32b-whisper
      groups:
      - minReplicas: 1
        name: default
        qosList:
        - free
      owner: no-reply@sambanova.ai
      secretNames:
      - sambanova-artifact-reader
    ```
  </Step>
</Steps>

To request new bundles, contact SambaNova support.

### Deploy multiple bundles

To serve more than one bundle at a time, apply a separate ModelDeployment for each bundle.

<Warning>
  SambaStack supports only one bundle per node. When deploying multiple bundles, assign each bundle to separate nodes to avoid resource conflicts. Use `spec.nodeSelector` on each ModelDeployment to control placement.
</Warning>

Verify that the pods reflect the deployed bundles:

<Tabs>
  <Tab title="Hosted">
    ```bash theme={}
    kubectl get pods
    ```
  </Tab>

  <Tab title="On Premise">
    ```bash theme={}
    kubectl -n <namespace> get pods
    ```
  </Tab>
</Tabs>

### Deploy a custom checkpoint

To serve your own converted or fine-tuned checkpoint, either override the checkpoint used by an existing model or register a new model for it. See [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints).

### Add a custom model architecture

Serving a model architecture that SambaStack does not yet support requires compiling a PEF and authoring `Pef`, `ModelProfile`, and `Model` resources. See [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints#add-a-custom-model-architecture).

### Update or remove a deployment or bundle

<Tabs>
  <Tab title="Update a configuration">
    <Steps>
      <Step title="Modify the YAML file">
        Edit the ModelBundle, ModelProfile, or ModelDeployment YAML file with your changes.
      </Step>

      <Step title="Reapply the configuration">
        <Tabs>
          <Tab title="Hosted">
            ```bash theme={}
            kubectl apply -f <modified-file>.yaml
            ```
          </Tab>

          <Tab title="On Premise">
            ```bash theme={}
            kubectl -n <namespace> apply -f <modified-file>.yaml
            ```
          </Tab>
        </Tabs>

        The legalizer automatically revalidates the changes. Changing a profile or a model triggers a reconcile of every bundle and deployment that references it.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Remove a configuration">
    <Steps>
      <Step title="Delete the ModelDeployment">
        <Tabs>
          <Tab title="Hosted">
            ```bash theme={}
            kubectl delete modeldeployment <deployment-name>
            ```
          </Tab>

          <Tab title="On Premise">
            ```bash theme={}
            kubectl -n <namespace> delete modeldeployment <deployment-name>
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Delete the ModelBundle (if one was used)">
        <Tabs>
          <Tab title="Hosted">
            ```bash theme={}
            kubectl delete modelbundle <bundle-name>
            ```
          </Tab>

          <Tab title="On Premise">
            ```bash theme={}
            kubectl -n <namespace> delete modelbundle <bundle-name>
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Delete the ModelProfile (optional)">
        Delete a profile only if you authored it for a custom architecture. Profiles provided with the SambaStack installation are shared across models and should be left in place.

        <Tabs>
          <Tab title="Hosted">
            ```bash theme={}
            kubectl delete modelprofile <profile-name>
            ```
          </Tab>

          <Tab title="On Premise">
            ```bash theme={}
            kubectl -n <namespace> delete modelprofile <profile-name>
            ```
          </Tab>
        </Tabs>
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Troubleshooting

### Legalizer validation failures

Errors appear in `status.legalizerInfo.errors` and cause `ValidationFailed`. Warnings appear in `status.legalizerInfo.warnings` and do not block deployment. See [Legalizer results](#legalizer-results).

| Error Pattern                                                                | Cause                                                                                    | Resolution                                                                                                                  |
| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `PEF pef1 and pef2 are not checkpoint compatible (checkpoint #0)`            | PEFs with the same `ckpt_sharing_uuid` cannot share checkpoint memory                    | Assign different `ckpt_sharing_uuid` values to the incompatible PEFs                                                        |
| `<checkpoint>: checkpoint sharing not enabled for <expert-a> and <expert-b>` | Warning only. The two experts cannot share checkpoint memory, so each holds its own copy | No action is required. To reclaim the duplicated memory, reduce the number of tiers or use PEFs that can share a checkpoint |
| `Bundle exceeds memory constraints`                                          | Combined PEF and checkpoint size exceeds RDU memory                                      | Reduce the number of tiers or batch sizes in the batching configuration                                                     |
| `PEF not found: <pef-name>`                                                  | Referenced PEF does not exist                                                            | Verify the PEF name with `kubectl get pefs`                                                                                 |
| Profile or model reference cannot be resolved                                | The named `ModelProfile` or `Model` does not exist in the namespace                      | Verify names with `kubectl get modelprofiles` and `kubectl get models`                                                      |

### Deployment failures

| Symptom                      | Possible Cause                                    | Resolution                                                                                   |
| ---------------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| Deployment stuck in pending  | Insufficient RDU resources                        | Check cluster capacity; reduce `minReplicas`                                                 |
| Checkpoint download fails    | Invalid checkpoint path or missing credentials    | Verify the `source` path in the Model; confirm the `sambanova-artifact-reader` secret exists |
| Model not accessible via API | Model name mismatch, or the model is not routable | Verify `spec.name` on the Model; confirm `modelSettings.routable` is not set to `false`      |
| Draft model served directly  | The draft model was left routable                 | Set `modelSettings.routable` to `false` on the draft model entry                             |

## Related documentation

<CardGroup cols={2}>
  <Card icon="rocket" href="/docs/en/sambastack/service-administration/model-deployment/bundles-v3" title="Migrating bundle configuration">
    Migrate bundles built with the deprecated custom resources
  </Card>

  <Card icon="cube" href="/docs/en/sambastack/service-administration/model-deployment/supported-models-and-bundles" title="Supported Models and Bundles">
    Catalogue of models and bundles available for deployment
  </Card>

  <Card icon="compress" href="/docs/en/sambastack/service-administration/model-deployment/deploy-custom-checkpoints" title="Custom checkpoint deployment">
    Deploy your own custom or fine-tuned checkpoints
  </Card>

  <Card icon="arrows-rotate" href="/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool" title="Checkpoint Conversion Tool">
    Convert Checkpoints to Compatible formats
  </Card>
</CardGroup>