SambaStack ModelDeployment Custom Resource

A ModelDeployment instantiates one or more replicas on the cluster and creates the serving pods behind a routable inference endpoint. It takes its models either from a ModelBundle or from an inline model and profile pair. ModelDeployment resources use apiVersion: sambanova.ai/v1alpha1.

Structure

Set exactly one model source:

Deployment path	Field	Description
Direct deployment	`spec.models`	Declares a model and profile inline. The specification is validated and legalized in place, and no bundle is used.
Bundle deployment	`spec.bundle`	References an existing `ModelBundle` by name. The bundle is trusted once it has been validated.

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120-llama70sd8-llama8
spec:
  bundle: mb-gpt120-llama70sd8-llama8
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200   # seconds; see Startup timeouts below

Field	Required	Description
`spec.bundle`	Conditional	Name of the `ModelBundle` to deploy. Set exactly one of `spec.bundle` or `spec.models`.
`spec.models`	Conditional	Inline definition of the models and profiles to deploy, using the same `modelConfigs` structure as a `ModelBundle`. Set exactly one of `spec.bundle` or `spec.models`.
`spec.groups[].name`	Yes	Name identifier for the deployment group. Use `default` unless you are defining more than one group.
`spec.groups[].minReplicas`	Yes	Minimum number of replicas to maintain.
`spec.groups[].qosList`	Yes	QoS tags that determine which service tiers can route requests to this group, for example `web`, `free`, or `pro`. The tags available to you are the ones defined by the service tiers configured on your system. See Service tiers.
`spec.owner`	Yes	Email address of the deployment owner for tracking and notifications.
`spec.secretNames`	Yes	Credentials used to access artifacts. Must match secrets configured in your environment.
`spec.engineConfig`	No	Configures the inference engine container: image, pull policy, environment variables, and its startup probe. See Startup timeouts.
`spec.cacheConfig`	No	Configures the artifact cache container, including its startup probe. See Startup timeouts.
`spec.tokenizerConfig`	No	Configures the tokenizer container, including its startup probe. See Startup timeouts.
`spec.storage`, `spec.storageClass`	No	Storage configuration for the deployment.
`spec.nodeSelector`, `spec.tolerations`	No	Default scheduling controls for the deployment.
`spec.skip_legalizer`	No	Skips the legalizer when deploying an inline `spec.models` specification. Spelled in snake case in the API, unlike the camel case fields above.

Startup timeouts

A deployment brings up three containers, each guarded by a Kubernetes startup probe. startupTimeout sets how long that probe keeps waiting before the container is considered failed, and startupInterval sets how often it polls. Both are in seconds.

Field	Guards	Default timeout	Default interval
`engineConfig.startupTimeout`	Inference engine container	`2700` (45 minutes)	`10`
`cacheConfig.startupTimeout`	Artifact cache container	`21600` (6 hours)	`1`
`tokenizerConfig.startupTimeout`	Tokenizer container	`60` (1 minute)	`1`

The timeout divided by the interval becomes the probe’s failure threshold, so raising the timeout extends the window rather than changing how often the container is checked.

When to raise engineConfig.startupTimeout

The engine container is ready only once every PEF and checkpoint in the deployment has been downloaded and loaded onto the RDUs. How long that takes scales with the number and size of the configurations being served, so a large bundle can outlast the 45-minute default and be killed mid-load. Setting startupTimeout: 7200 gives the engine two hours, which is enough for the large model bundles SambaNova provides. This is why the examples throughout these pages use that value:

spec:
  engineConfig:
    startupTimeout: 7200   # seconds; 2 hours, enough for large bundles to load

Raising the timeout costs nothing when startup is quick — the probe succeeds as soon as the container is ready and does not wait out the remaining time. A deployment stuck in a restart loop while loading a large bundle is the signal that the value is too low.

Deploy a single model

Deploy with an inline spec.models specification

Deploy a bundle

Deploy with a spec.bundle reference

Get started

Models

Features

Build

Resources

ModelDeployment

Structure

Startup timeouts

When to raise engineConfig.startupTimeout

Deploy a single model

Deploy a bundle

​Structure

​Startup timeouts

​When to raise engineConfig.startupTimeout

​Related resources

Deploy a single model

Deploy a bundle

Structure

Startup timeouts

When to raise engineConfig.startupTimeout

Related resources