SambaStack Prompt Caching - SambaNova Documentation

Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times. Prompt caching is available for MiniMax-M2.7 on SambaStack 1.2.

This page covers how to use prompt caching and interpret usage fields. For model availability, see Supported models and bundles.

When to use prompt caching

Prompt caching delivers the most savings when a large portion of your input is repeated across requests:

Good fit	Poor fit
Long system prompt shared across all user turns	Fully unique inputs per request (no shared prefix)
Repeated few-shot examples or tool definitions	Very short prompts where caching overhead outweighs savings
RAG context re-sent with each query	Single-use documents where the prefix never repeats
Multi-turn conversations with a fixed preamble

If cached_tokens stays at 0 across requests, your inputs do not share a cacheable prefix – review whether your system prompt or context block is stable across calls.

How prompt caching works

Automatic Prefix Caching (APC) is available for MiniMax-M2.7 but must be enabled by a SambaStack administrator by deploying the dyt-minimax-m2p7-32-64-192k-pc bundle. Once enabled, APC detects shared prefixes across requests and reuses cached computations transparently – no changes to your API requests are required. Prompt caching follows the OpenAI standard. Cache hits are detected automatically and reported in the response usage field – no cache_control markers or other request body changes are needed. APC is local to each serving instance. Cached KV state is stored in local DDR and is not transferred across instances. On multi-instance deployments, your prefix is cached independently on each instance after it is served there. The sequence state pool size for MiniMax-M2.7 is 192K.

Checking cache usage in the response

Every /v1/chat/completions response from a cache-enabled model includes cache token counts in the usage field. The following example shows a second request that shares the same long system prompt as a prior request – the prefix is served from cache rather than reprocessed. Request (second call – cache warm)

# A long, repeated system prompt (1,000+ tokens) triggers APC on the second call.
curl https://<sambastack-host>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMax-M2.7",
    "messages": [
      {
        "role": "system",
        "content": "You are a technical assistant for SambaNova hardware. ..."
      },
      {
        "role": "user",
        "content": "What is the default batch size for MiniMax-M2.7?"
      }
    ],
    "max_tokens": 100
  }'

Response

{
  "id": "cmpl-3ca55e7f-8fd5-470d-bac6-ee38a7a5307b",
  "object": "chat.completion",
  "created": 1779835973,
  "model": "MiniMax-M2.7",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The default batch size for MiniMax-M2.7 is 2."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5797,
    "completion_tokens": 100,
    "total_tokens": 5897,
    "prompt_tokens_details": {
      "cache_creation_tokens": 0,
      "cached_tokens": 4096
    }
  }
}

Field	Description
`cached_tokens`	Input tokens served from the cache. These are billed at the cached input rate.
`cache_creation_tokens`	Input tokens used to populate the cache on this request. Informational only – no additional charge.

Input remaining = prompt_tokens − cached_tokens. This portion is billed at the standard input rate. A cached_tokens value of 0 means no cache hit occurred on that request – it is not an error. This happens on the first request for a given prefix (cache is being populated) or when the input does not share a prefix with any cached entry. For models that do not support prompt caching, cached_tokens is always 0 and billing is unchanged.

Code example

from sambanova import SambaNova

client = SambaNova(
    base_url="https://<sambastack-host>",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
the following reference documentation:

[Insert your long, stable system prompt or document here – the longer and more stable this
prefix, the more tokens are eligible for caching. A prefix must be at least 4,096 tokens
to qualify.]"""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What is the default batch size for MiniMax-M2.7?")

# Second request – same prefix, served from cache
ask("What is the maximum sequence length for MiniMax-M2.7?")

from openai import OpenAI

client = OpenAI(
    base_url="https://<sambastack-host>/v1",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
the following reference documentation:

[Insert your long, stable system prompt or document here – the longer and more stable this
prefix, the more tokens are eligible for caching. A prefix must be at least 4,096 tokens
to qualify.]"""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What is the default batch size for MiniMax-M2.7?")

# Second request – same prefix, served from cache
ask("What is the maximum sequence length for MiniMax-M2.7?")

Billing impact

Cached tokens are billed at a lower rate than standard input tokens. The billing calculation for a cache-enabled request is:

Token type	Billing
Input remaining (`prompt_tokens` − `cached_tokens`)	Standard input rate
`cached_tokens`	Cached input rate (lower)
`completion_tokens`	Standard output rate (unchanged)

For example, a request with 5,797 total input tokens where 4,096 are served from cache: you pay the standard input rate on 1,701 tokens and the cached input rate on 4,096 tokens. Output billing is unchanged. cache_creation_tokens appears in the response but is informational only – it represents tokens used to populate the cache on this request and carries no additional charge at this time. For models without prompt caching, cached_tokens is always 0, so the formula reduces to the standard calculation with no billing change.

Contact your SambaNova representative for current pricing, including the cached input rate for MiniMax-M2.7.

Limitations

Prompt caching is available on MiniMax-M2.7 only in SambaStack 1.2.
Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.
APC operates on shared prefixes. Requests that diverge before the cached boundary do not get a cache hit.
The maximum cached prefix (sequence state) is 192K tokens.
Cache eviction uses an LRU (least recently used) policy. Cache persistence duration varies with system load and is not guaranteed.
A prefix must contain at least 4,096 tokens to qualify for caching.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Using prompt caching

When to use prompt caching

How prompt caching works

Checking cache usage in the response

Code example

Billing impact

Limitations

​When to use prompt caching

​How prompt caching works

​Checking cache usage in the response

​Code example

​Billing impact

​Limitations

When to use prompt caching

How prompt caching works

Checking cache usage in the response

Code example

Billing impact

Limitations