> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Choose a deployment configuration

SambaStack supports two deployment configurations for supported models: **high-interactivity** and **high-throughput**. Both use the same model weights and the same API – they differ only in how the system handles requests. Use high-interactivity for low-latency, user-facing applications. Use high-throughput for batch or high-concurrency workloads where aggregate output matters more than per-user latency. The rest of this page covers the trade-offs, the PEF configurations for each, and how to build a bundle.

> [!NOTE]
>
> High-throughput and high-interactivity configurations require dedicated systems. Models deployed in either configuration cannot be bundled with other models. If you are unfamiliar with bundles or bundle templates, see [Deploying model bundles](https://docs.sambanova.ai/docs/en/v1.0.57/sambastack/service-administration/deploying-model-bundles.md).

## Deployment configurations

| Configuration | Profile | Best when |
| :- | :- | :- |
| **High-throughput** | Aggregate token throughput across concurrent requests | Batch processing, asynchronous workloads, or large user volumes where total system output matters more than per-user latency |
| **High-interactivity** | Per-request latency and time-to-first-token | Real-time, user-facing applications |

> [!NOTE]
>
> Both configurations use the same model name in API calls. The same request works against either configuration:
>
> ```json theme={}
> { "model": "DeepSeek-R1", "messages": [...] }
> ```
>
> The configuration controls request handling on the server side; no client-side changes are required.

## When to use each configuration

### High-throughput

Use the high-throughput configuration when:

* You are serving many concurrent users and aggregate throughput matters more than per-user latency
* Your workload is asynchronous or batch-oriented (for example, document processing or offline inference pipelines)
* End-to-end latency per request is not a constraint

### High-interactivity

Use the high-interactivity configuration when:

* You are building real-time, user-facing applications
* Per-user time-to-first-token and tokens-per-second are the primary metrics
* Your deployment has fewer nodes, or your users have tight latency budgets

## Supported models

Both configurations are available for the following models:

* DeepSeek-R1
* DeepSeek-V3-0324
* DeepSeek-V3.1
* DeepSeek-V3.1-Terminus
* DeepSeek-V3.2

## Architecture

The high-throughput configuration uses continuous batching, separating the prefill and decode phases into a dedicated pipeline. Two modes are available:

* **Aggregated (ACB):** Prefill and decode run collocated on the same nodes.
* **Disaggregated (DCB):** Prefill and decode run on separate dedicated nodes, so each phase can be sized independently. The recommended node split is more prefill nodes than decode nodes – for example, three prefill nodes and one decode node.

> [!WARNING]
>
> DCB has not yet been internally validated on SambaStack. Use ACB for SambaStack deployments until DCB validation is published.

* **Prefill nodes** process the input prompt
* **Decode nodes** generate output tokens

> [!WARNING]
>
> The high-throughput configuration requires a minimum of 4 nodes in disaggregated mode. For single-node or small deployments, use the high-interactivity configuration instead.

## Requirements and limitations

| Constraint | Details |
| :- | :- |
| **Minimum nodes (high-throughput)** | 4 nodes in a multinode setup |
| **Dedicated systems** | High-throughput and high-interactivity configurations require dedicated systems – bundling with other models is not supported |
| **Node configuration (disaggregated mode)** | More prefill nodes than decode nodes required (for example, 3 prefill : 1 decode) |
| **Checkpoint version** | Use the latest checkpoint version listed in the Model CR – older versions are not compatible with high-throughput PEFs |

## PEF configurations

Use the following PEF CR identifiers when building your bundles. See [Custom Bundle Deployment](https://docs.sambanova.ai/docs/en/v1.0.57/sambastack/service-administration/custom-bundle-deployment.md) for the full bundle-building procedure.

> [!NOTE]
>
> **Custom Resource (CR):** A Kubernetes extension object. Model CRs and PEF CRs define model and PEF configurations in the cluster.

> [!NOTE]
>
> When referencing a PEF CR in a BundleTemplate, append the version number: for example, `deepseek-ss8192-bs1:1`. Use version `1` unless `kubectl describe pef <pef-name>` shows a higher stable version is available.

### High-throughput PEFs

| PEF CR | Sequence length | Batch size |
| :- | :- | :- |
| `deepseek-ss32768-bs1-cb2-64` | 32768 (32K) | 64 |
| `deepseek-ss16384-bs1-cb2-128` | 16384 (16K) | 128 |
| `deepseek-ss8192-bs1-cb2-256` | 8192 (8K) | 256 |

**Picking a high-throughput PEF:**

* Choose the sequence length (`ss`) that fits your longest prompt plus expected output tokens.
* Higher batch sizes serve more concurrent decode requests per node but require more RDU memory. The table lists the supported combinations.

### High-interactivity PEFs

| PEF CR | Sequence length | Batch size |
| :- | :- | :- |
| `deepseek-ss4096-bs1` | 4096 (4K) | 1 |
| `deepseek-ss4096-bs4` | 4096 (4K) | 4 |
| `deepseek-ss8192-bs1` | 8192 (8K) | 1 |
| `deepseek-ss8192-bs4` | 8192 (8K) | 4 |
| `deepseek-ss16384-bs1` | 16384 (16K) | 1 |
| `deepseek-ss32768-bs1` | 32768 (32K) | 1 |
| `deepseek-ss131072-bs1` | 131072 (128K) | 1 |

**Picking a high-interactivity PEF:**

* Match the sequence length to your prompt plus expected output budget.
* `bs1` minimizes per-user latency. `bs4` trades a small latency increase for higher per-node throughput when you have multiple concurrent users.

## Build a bundle

No prebuilt bundles ship for these configurations – you create a custom bundle using the PEF CRs listed above. Follow the [Custom Bundle Deployment](https://docs.sambanova.ai/docs/en/v1.0.57/sambastack/service-administration/custom-bundle-deployment.md) guide and reference the relevant PEF CR when defining your BundleTemplate.

When using a high-throughput PEF in your BundleTemplate, set `continuous_batching: true` in the expert definition:

```yaml theme={}
DeepSeek-V3-0324:
  experts:
    8k:
      configs:
      - continuous_batching: true
        pef: deepseek-ss8192-bs1-cb2-256:1
```

Then configure your BundleDeployment for the appropriate mode.

**Aggregated mode (ACB):**

```yaml theme={}
groups:
  - name: default
    continuous_batching:
      mode: aggregate
    minReplicas: 1
    qosList:
    - web
    - free
```

**Disaggregated mode (DCB):**

```yaml theme={}
groups:
  - name: default
    continuous_batching:
      use_mpi: true
      prefill:
        minReplicas: 3
      decode:
        minReplicas: 1
    minReplicas: 1
    qosList:
    - web
    - free
```

## Verify your deployment

After deploying the bundle, confirm the configuration is active:

```shellscript theme={}
kubectl describe bundledeployment <bundle-deployment-name>
```

In the output, look for `continuous_batching.mode` set to `aggregate` (ACB) or `disaggregate` (DCB), and confirm the replica counts under `prefill` and `decode` match what you configured.

## Switch between configurations

To switch between high-throughput and high-interactivity, redeploy the bundle with the appropriate PEF and BundleTemplate settings. When switching to high-interactivity, remove `continuous_batching: true` from the expert and remove the `continuous_batching` block from the BundleDeployment. The model name in API calls does not change.

## Monitor your deployment

The SambaStack logging system emits per-request metrics relevant to these deployments:

| Metric | Log key | What it tells you |
| :- | :- | :- |
| Decode queue time | `decode_queue_time` | Time spent waiting in continuous batching queues – high values indicate decode saturation |
| Time to first token | `time_to_first_token` | Prefill latency per request – key indicator for high-interactivity deployments |
| Completion tokens/sec | `completion_tokens_per_sec` | Aggregate throughput – key indicator for high-throughput deployments |

See [Logs](https://docs.sambanova.ai/docs/en/v1.0.57/sambastack/reference-architecture/observability/logs.md) for the full list of available metrics and example queries.