SambaStack Custom Checkpoint Deployment

In SambaStack, you can deploy your own custom or fine-tuned checkpoints for inference in the same manner as deploying standard model offerings, with a few additional steps to prepare your checkpoint for use in the platform. Once prepared and deployed, custom checkpoints behave just like any other checkpoint you deploy on SambaStack. This page also documents the Model and Pef resources, which you author when bringing your own checkpoint or model architecture.

Overview

Deploying a custom checkpoint involves four high-level actions:

Convert your checkpoint into a SambaNova-compatible format using the Checkpoint Conversion Tool.
Upload your converted checkpoint to your private Google Cloud Storage bucket configured with read permissions granted to your SambaNova-provided service account OR make it available in NFS such that it is readable by your cluster.
Reference your checkpoint, either by overriding the checkpoint used by an existing model or by registering a new Model resource for it.
Deploy it by pairing the model with a compatible ModelProfile in a ModelDeployment or a ModelBundle.

Before starting this workflow, ensure you have completed the checkpoint conversion process. See the Checkpoint Conversion Tool page for instructions.

Prerequisites

Before deploying a custom checkpoint, ensure you have:

A converted checkpoint in SambaNova-compatible format (see Checkpoint Conversion Tool)
Your NFS mounted storage or access to a Google Cloud Storage (GCS) bucket
Your SambaNova-provided service account JSON file
kubectl configured with access to your SambaStack cluster
Familiarity with deploying models and bundles, including model profiles and model bundles

Supported models for custom checkpoints

Custom checkpoint deployment is supported for a growing set of base models in SambaStack. See the Supported Models and Bundles table to find models that support custom checkpoints.

Steps to deploy a custom checkpoint

Convert your checkpoint

Custom or fine-tuned checkpoints must be converted into a format optimized for SambaNova’s SN40L hardware before they can be deployed. SambaNova provides a Checkpoint Conversion Tool, delivered as a Docker container that you can run locally. The tool generates converted checkpoint artifacts that can then be uploaded and deployed for inference on SambaStack.To begin, follow the instructions in the Download and set up section of the Checkpoint Conversion Tool documentation. Setup is complete once you have downloaded the conversion tool container and synced the model metadata with your specific SambaStack instance.After setup, use the steps described in the Convert and validate checkpoint section of the Checkpoint Conversion Tool documentation to convert your custom checkpoint into the SambaNova-compatible format.

Configure GCS bucket permissions

You can skip this section if you have NFS mounted to your cluster.

SambaStack uses Google Cloud Storage (GCS) to store checkpoints and other SambaStack artifacts. For custom checkpoints, you’ll store the converted checkpoint artifacts in your own GCS bucket. To make these artifacts available to SambaStack during deployment, your SambaNova-provided service account needs read access to your bucket.

This is a one-time setup step. After permissions are in place, you can upload any number of custom checkpoints to your bucket and use them directly in your deployments.

Identifying your service account

Your service account information is provided as a JSON file. Locate the client_email field - this is the identity that needs read access to your bucket. For example:

{
  "type": "service_account",
  "project_id": "example-project-id",
  "private_key_id": "example-private-key-id",
  "private_key": "-----BEGIN PRIVATE KEY-----\n<private key contents>\n-----END PRIVATE KEY-----\n",
  "client_email": "ss-artifacts-reader@example-project-id.iam.gserviceaccount.com",
  "client_id": "12345678901234567890",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ss-artifacts-reader%40example-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Granting Storage Object Viewer role

To allow SambaStack to access your custom checkpoints, grant the service account the Storage Object Viewer role on your bucket. This provides read-only access to objects without allowing writes or modifications.Using the Google Cloud Console:

Open the Google Cloud Console.
Navigate to Storage → Buckets, and select the bucket you plan to use.
Go to the Permissions tab.
Click + Add principal.
In the New principals field, enter your service account’s client_email.
In the Role dropdown, choose: Cloud Storage → Storage Object Viewer
Click Save.

Using the gcloud CLI:Before running the commands below, identify:

<BUCKET_NAME> – the name of your GCS bucket
<SERVICE_ACCOUNT_EMAIL> – the client_email value from your service account JSON
<PROJECT_ID> – the Google Cloud project that owns the bucket

To grant the Storage Object Viewer role at the bucket level:

gcloud storage buckets add-iam-policy-binding gs://<BUCKET_NAME> \
    --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
    --role="roles/storage.objectViewer" \
    --project=<PROJECT_ID>

To verify that the role was successfully applied:

gcloud storage buckets get-iam-policy gs://<BUCKET_NAME> \
    --project=<PROJECT_ID>

You should see an entry resembling:

bindings:
- members:
  - serviceAccount:<SERVICE_ACCOUNT_EMAIL>
  role: roles/storage.objectViewer

For additional guidance, see Google’s IAM documentation:

Upload your converted checkpoint

NFS

If you have NFS mounted to your cluster, verify that the converted checkpoint is moved to NFS and accessible by your cluster.

Google Cloud Storage

If you are using Google Cloud Storage (GCS), after converting the checkpoint, upload the directory containing the converted checkpoint files to your GCS bucket.

This step may take a while depending on the size of your checkpoint.

Using the Google Cloud Console:

Open the Google Cloud Console.
Navigate to Storage → Buckets and select the bucket you’ve configured for custom checkpoints.
Click Upload folder (or Upload files, depending on your structure).
Select the directory containing your converted checkpoint artifacts.
Wait for the upload to complete; the structure should remain intact.

Using the gcloud CLI:You can upload the entire converted checkpoint directory recursively with:

gcloud storage cp -r <LOCAL_CONVERTED_CHECKPOINT_DIR> gs://<BUCKET_NAME>/<DESTINATION_PREFIX>/

Reference your checkpoint

There are two ways to make your checkpoint servable, depending on whether you want to keep the existing model name or serve the checkpoint under a new one.

Option 1: Override the checkpoint of an existing model

Use this option to reuse an existing model’s name and serve your checkpoint in place of the one that model normally uses. No new Model resource is required. Set checkpointOverrides on the model configuration in your ModelBundle or ModelDeployment:

spec:
  modelConfigs:
  - model: meta-llama-3-1-8b-instruct:1
    profile: llama-3p1-8b
    modelSettings:
      checkpointOverrides:
        checkpoint:
          source: gs://<BUCKET_NAME>/path/to/converted/checkpoint
          checkpoint_status: stable
          tool_support: false

Requests continue to use the existing model’s serving name. This is the recommended approach for serving a fine-tuned variant of a supported model.

Option 2: Register a new Model resource

Use this option to serve the checkpoint under its own model name, for example a checkpoint fine-tuned from another model that you want addressed separately in the inference API. This requires creating a new Model resource.Set spec.checkpoints.<arch> using an architecture key compatible with the profile you intend to pair it with, set source to the location of your converted checkpoint, and set spec.tokenizer.path to the tokenizer of the base model the checkpoint was derived from.

apiVersion: sambanova.ai/v1alpha1
kind: Model
metadata:
  name: my-custom-llama-3-1-8b   # must be a valid Kubernetes resource name
spec:
  name: my-custom-llama-3-1-8b   # serving name used in the inference API
  owner: jane@doe.ai
  public: true
  aliases:
  - My-Custom-Llama3.1-8B
  checkpoints:
    llama3:
      versions:
        "1":
          source: gs://<BUCKET_NAME>/path/to/converted/checkpoint
          checkpoint_status: stable
          tool_support: false
  tokenizer:
    path: ./Meta-Llama-3.1-8B-Instruct_tokenizer
  metadata:
    capabilities:
    - text

The architecture key in this example, llama3, is illustrative. Use a key that matches the model_arch of the profile you intend to use. For the full field reference, see Model structure.

The tokenizer field is used only for checking inputs to calculate sequence length requirements prior to generation time. Set it to the tokenizer of the base model your checkpoint was fine-tuned from.

Apply the resource:

Hosted
On Premise

kubectl apply -f <model-file>.yaml

kubectl -n <namespace> apply -f <model-file>.yaml

Deploy the checkpoint

Pair the model with a ModelProfile whose model_arch is compatible with your checkpoint’s architecture, then deploy it either directly or through a bundle.

Compatibility between a checkpoint and a profile is determined by model_arch, and the operator does not verify it for you. Select a profile whose architecture matches the checkpoint you are serving. A mismatch results in inference errors.

To find a compatible profile, list the profiles in your cluster and note the model_arch of each:

Hosted
On Premise

kubectl get modelprofiles

kubectl -n <namespace> get modelprofiles.sambanova.ai

Then deploy. For a single model, inline it in a ModelDeployment:

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-my-custom-llama-3-1-8b
spec:
  models:
    modelConfigs:
    - model: my-custom-llama-3-1-8b:1
      profile: llama-3p1-8b
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: jane@doe.ai
  secretNames:
  - sambanova-artifact-reader

Apply it:

Hosted
On Premise

kubectl apply -f <modeldeployment-file>.yaml

kubectl -n <namespace> apply -f <modeldeployment-file>.yaml

For the full deployment options, including bundling several models together, see Deploying models and bundles.

After your deployment is running, use the serving name you defined (for example, my-custom-llama-3-1-8b) in your inference API requests.

Resource reference

The following resources are the ones you author when bringing your own checkpoint or model architecture. For the profile, bundle, and deployment resources, see Deploying models and bundles.

Model structure

A Model resource is the source of checkpoints. It holds the checkpoint versions for each architecture the model supports, keyed by architecture name, along with the tokenizer to use.

apiVersion: sambanova.ai/v1alpha1
kind: Model
metadata:
  name: deepseek-v3-2
spec:
  name: DeepSeek-V3.2
  owner: no-reply@sambanova.ai
  checkpoints:
    deepseek:
      versions:
        "1":
          source: gs://<SAMBASTACK_ARTIFACTS_BUCKET>/path/to/checkpoint
          checkpoint_status: stable
          tool_support: true
  tokenizer:
    path: ./DeepSeek-V3.2_tokenizer
  metadata:
    capabilities:
      - text

Field	Required	Description
`metadata.name`	Yes	Kubernetes resource name. This value is used in `modelConfigs[].model` references.
`spec.name`	Yes	Serving name used in the inference API.
`spec.aliases`	No	Alternative names that route to this model.
`spec.checkpoints.<arch>`	Yes	Checkpoints keyed by architecture. The `<arch>` key must match the `model_arch` value of the PEFs in the profile that this model is paired with.
`spec.checkpoints.<arch>.versions.<version>.source`	Yes	Path pointing to the model checkpoint. Find available checkpoints in Supported Models and Bundles.
`spec.checkpoints.<arch>.versions.<version>.checkpoint_status`	Yes	Lifecycle status of this checkpoint version. See PEF and checkpoint lifecycle status.
`spec.checkpoints.<arch>.versions.<version>.tool_support`	No	Whether this checkpoint is compatible with tools and function calling. Defaults to `false`.
`spec.checkpoints.<arch>.versions.<version>.vision_embedding_checkpoint`	No	Companion vision embedding checkpoint, used for multimodal models.
`spec.tokenizer.path`	Yes	Path to the local tokenizer to use. For a custom checkpoint, set this to the tokenizer of the base model that the checkpoint was derived from.
`spec.tokenizer.endpointUrls`	No	URLs to send tokenizer requests to. One URL is selected at random per request.
`spec.owner`	Yes	Email address of the model owner for tracking and notifications.
`spec.tool_support`	No	Whether the model supports using tools.
`spec.public`	No	Whether the model appears in the public models list.
`spec.metadata.capabilities`	Yes	Capabilities of the model, for example `text`, `vision`, `audio`, `embeddings`, or `reasoning`.
`spec.metadata`	No	Additional catalog metadata surfaced in the models API and documentation: `provider`, `license`, `overview`, `architecture`, `languages`, `category`, `text_only`, and `vocabulary_size`.
`spec.expertFields`	No	Defaults applied to every expert instance of this model at deployment time: `output_processor`, `postprocess_parser`, `enable_reasoning_effort`, and `tiktoken_vocab_rel_path`. Bundle-side values for the same keys take precedence.
`spec.price`	No	Pricing for the model: `input_tokens`, `output_tokens`, `input_cache_read`, `input_cache_write`, and `input_audio_duration_per_hour`.
`spec.fallback`	No	Fallback configuration, typically for multimodal support. Takes `model` and `when_missing`, both required when set.
`spec.api_forward_url`	No	Forwards traffic for this model to another cluster.

The paths to checkpoints hosted by SambaNova will be provided to you by your SambaNova contact. If you have hosted your own checkpoints, you can include those paths in the source fields above.

PEF structure

A Pef resource registers a compiled executable. The profile determines which PEF, sequence size, and batch size are used, so you select a profile rather than an individual PEF. Author Pef resources only when introducing a new model architecture.

Field	Required	Description
`spec.metadata.model_arch`	Yes	Architecture of this PEF, which determines the checkpoints that the PEF is compatible with.
`spec.metadata.batch_size`	Yes	Batch size that the PEF was compiled for.
`spec.metadata.max_seq_length`	Yes	Maximum sequence length that the PEF supports.
`spec.metadata.seq_lengths`	No	Sequence lengths that the PEF supports. The maximum value must equal `max_seq_length`.
`spec.metadata.num_rdus`, `spec.metadata.rdu_arch`	No	Number of RDUs and the RDU architecture that the PEF was compiled for, for example `sn40`.
`spec.metadata.continuous_batching`, `spec.metadata.use_context_cache`, `spec.metadata.constrained_decoding`, `spec.metadata.is_spec_prefill`	No	Feature support flags. These values determine the `features` list of the profiles built from this PEF.
`spec.metadata.dynamic_dims`	No	Dynamic dimensions available in this PEF, such as `batch_size` and the sequence length dimensions.
`spec.versions.<version>.source`	Yes	Location to download this version of the PEF from.
`spec.versions.<version>.cached_path`	Yes	Path to this PEF as cached in the legalizer database.
`spec.versions.<version>.ckpt_sharing_uuid`	Yes	Identifier that defines checkpoint-sharing relationships. PEFs with the same identifier can share a checkpoint.
`spec.versions.<version>.pef_status`	Yes	Lifecycle status of this PEF version. See PEF and checkpoint lifecycle status.
`spec.owner`	Yes	Owner of the PEF.
`spec.secretName`	Yes	Name of the single secret required in order to download the artifact.

PEF and checkpoint lifecycle status

SambaStack assigns a pef_status field to PEF CR versions and a checkpoint_status field to model CR checkpoint versions to indicate their support lifecycle. Understanding these statuses helps you make informed decisions when selecting PEF or checkpoint versions. PEF and checkpoint version status values Each version entry in a PEF CR includes a pef_status field. Model CR checkpoint versions use checkpoint_status. Both share the same set of values:

Status	Description
`preview`	Not fully tested or supported. May have unknown reliability or performance issues, or limited functionality (for example, partial function calling support). Not recommended for production workloads.
`stable`	Fully supported and tested.
`deprecated`	Has known reliability or performance issues. Still available for a limited transition period (up to 3 months from the deprecation announcement) to allow migration to a stable version.
`removed`	No longer usable. The version entry is retained in the PEF CR or model CR for traceability and auditability, but the path may no longer exist, causing deployment to fail if referenced.

Example PEF CR versions with status

versions:
  '1':
    source: gs://ext-sambastack-artifacts-prod-0/path/to/pef_v1.pef
    pef_status: deprecated
  '2':
    source: gs://ext-sambastack-artifacts-prod-0/path/to/pef_v2.pef
    pef_status: stable

To check version statuses, run kubectl describe pef <pef-name> or kubectl describe model <model-name> and review the pef_status or checkpoint_status field in the Versions section.

The following procedures describe the step-by-step workflow for creating and deploying custom deployments and bundles using the concepts and structures described above.

Add a custom model architecture

If the model architecture is not yet supported, no compiled executable and no profile exist for it, so the full set of resources must be authored in the following order:

Order	Resource	Purpose
1	`Pef`	Registers a PEF compiled for the new architecture, including its `spec.metadata.model_arch`, batch size, maximum sequence length, and feature support flags.
2	`ModelProfile`	Lists the new PEFs in `spec.pefs`, surfaces the new `model_arch` value, and defines the default batching support for the architecture.
3	`Model`	Holds the checkpoint sources under the same architecture key as the PEFs, together with the tokenizer path.
4	`ModelDeployment`	Deploys the model, either directly or through a `ModelBundle`.

A new PEF must be compiled for the architecture, and a Pef resource must be created to register it. Compiling a PEF is not covered on this page. Use the resources for the closest supported model as a reference for the values in each resource:

Hosted
On Premise

kubectl get pef <pef-name> -o yaml
kubectl get modelprofile <profile-name> -o yaml
kubectl get model <model-name> -o yaml

kubectl -n <namespace> get pef.sambanova.ai <pef-name> -o yaml
kubectl -n <namespace> get modelprofile.sambanova.ai <profile-name> -o yaml
kubectl -n <namespace> get model.sambanova.ai <model-name> -o yaml

Verifying your deployment

After applying the deployment, verify that your custom checkpoint deployment is successful:

Check deployment status:

kubectl get modeldeployments
kubectl describe modeldeployment <your-deployment-name>

Verify the model is available:
kubectl get models
Test with a sample inference request using your custom model name. See the Quickstart Guide for Developers for example inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL.

Troubleshooting

Common issues

Issue	Possible Cause	Solution
Deployment fails with permission error	Service account lacks read access to GCS bucket	Verify Storage Object Viewer role is granted (see Step 2)
Model not found in API requests	Model name mismatch, or the model is not routable	Ensure requests use `spec.name` from the `Model` resource, and that `modelSettings.routable` is not set to `false`
Checkpoint files not found	Incorrect checkpoint path	Verify the `source` path matches your uploaded checkpoint location
Inference errors	Checkpoint incompatible with the profile	Ensure the profile’s `model_arch` is compatible with your checkpoint’s architecture

Verifying GCS access

If you suspect permission issues, verify that your service account can access the checkpoint:

gcloud auth activate-service-account --key-file=<path-to-service-account-json>
gsutil ls gs://<BUCKET_NAME>/<CHECKPOINT_PATH>/

Next steps

To deploy custom checkpoints with speculative decoding, see Deploying with speculative decoding
For monitoring and observability, see SambaStack Monitoring
If your custom checkpoint uses a different chat template or tool-call output format than the base model, see Custom chat templates and output parsing in the Developer Guide for how to handle prompt formatting and parsing on the client side.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Deploying custom checkpoints

Overview

Prerequisites

Supported models for custom checkpoints

Steps to deploy a custom checkpoint

Convert your checkpoint

Configure GCS bucket permissions

Identifying your service account

Granting Storage Object Viewer role

Upload your converted checkpoint

NFS

Google Cloud Storage

Reference your checkpoint

Option 1: Override the checkpoint of an existing model

Option 2: Register a new Model resource

Deploy the checkpoint

Resource reference

Model structure

PEF structure

PEF and checkpoint lifecycle status

Add a custom model architecture

Verifying your deployment

Troubleshooting

Common issues

Verifying GCS access

Next steps

​Overview

​Prerequisites

​Supported models for custom checkpoints

​Steps to deploy a custom checkpoint

Convert your checkpoint

Configure GCS bucket permissions

​Identifying your service account

​Granting Storage Object Viewer role

Upload your converted checkpoint

​NFS

​Google Cloud Storage

Reference your checkpoint

​Option 1: Override the checkpoint of an existing model

​Option 2: Register a new Model resource

Deploy the checkpoint

​Resource reference

​Model structure

​PEF structure

​PEF and checkpoint lifecycle status

​Add a custom model architecture

​Verifying your deployment

​Troubleshooting

​Common issues

​Verifying GCS access

​Next steps

Overview

Prerequisites

Supported models for custom checkpoints

Steps to deploy a custom checkpoint

Identifying your service account

Granting Storage Object Viewer role

NFS

Google Cloud Storage

Option 1: Override the checkpoint of an existing model

Option 2: Register a new Model resource

Resource reference

Model structure

PEF structure

PEF and checkpoint lifecycle status

Add a custom model architecture

Verifying your deployment

Troubleshooting

Common issues

Verifying GCS access

Next steps