Prompt Caching Guide - SambaNova Documentation

Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times.

Prompt caching is available on MiniMax-M2.7 on SambaCloud. Automatic Prefix Caching (APC) is enabled by default – no changes to your request are required.

When to use prompt caching

Prompt caching delivers the most savings when a large portion of your input is repeated across requests:

Good fit	Poor fit
Long system prompt shared across all user turns	Fully unique inputs per request (no shared prefix)
Repeated few-shot examples or tool definitions	Very short prompts where savings are negligible
RAG context re-sent with each query	Single-use documents where the prefix never repeats
Multi-turn conversations with a fixed preamble

If cached_tokens stays at 0 across requests, your inputs do not share a cacheable prefix – check whether your system prompt or context block is stable across calls.

How it works

Automatic Prefix Caching (APC) is always on for MiniMax-M2.7. It detects shared prefixes across requests and reuses cached computations transparently – no API changes required. Keep the leading portion of your messages stable and consistent across requests to maximize cache hits. A prefix is matched when the leading token sequence of the messages array is identical to a cached request. Any change to the prefix – rewording the system prompt, inserting a message before it, or reordering content – starts a new cache entry. Changing only the user message while keeping the system prompt identical reuses the cached prefix. Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.

Read cache usage from the response

Every response includes a prompt_tokens_details object showing how many tokens were served from cache:

{
  "usage": {
    "prompt_tokens": 5797,
    "completion_tokens": 100,
    "total_tokens": 5897,
    "prompt_tokens_details": {
      "cache_creation_tokens": 0,
      "cached_tokens": 4096
    }
  }
}

Field	Description
`cached_tokens`	Tokens served from cache. Billed at the cached input rate (lower than standard).
`cache_creation_tokens`	Tokens written to cache on this request. Informational only – no additional charge.

cache_creation_tokens is non-zero on the first request that builds the cache entry and 0 on subsequent cache hits. The number of non-cached input tokens is prompt_tokens - cached_tokens.

Billing

Cached tokens are billed at a lower rate than standard input tokens. The full pricing for your model – including the cached input rate and cache write rate – is available via the /v1/models endpoint and on the SambaCloud pricing page.

curl https://cloud.sambanova.ai/v1/models \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  | jq '.data[] | select(.id == "MiniMax-M2.7") | .pricing'

The billing formula for a cache-enabled request:

cost = (prompt_tokens − cached_tokens) × standard_input_rate
     + cached_tokens × cached_input_rate
     + completion_tokens × output_rate

For example, a request with 5797 total input tokens where 4096 are served from cache: only 1701 tokens are billed at the standard input rate, and 4096 at the lower cached rate. Output billing is unchanged.

Code example

Caching activates only when the shared prefix reaches 4096 tokens. The system prompt below is shorter for readability – replace it with your actual document to observe cache hits in practice.

from sambanova import SambaNova

client = SambaNova(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a financial analyst assistant. You have access to the following
quarterly earnings report:

Fiscal Q3 2024 Earnings Report – Your Company

Revenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenue
was $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).

Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross margin
was 71.1%.

Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operating
income was $1.05 billion (25.0% margin).

Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net income
was $910 million, or $1.82 per diluted share.

Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.
Free cash flow was $980 million.

Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.

Replace this content with your actual document. The longer and more stable your system prompt,
the more tokens are eligible for caching."""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What was the revenue in Q3?")

# Second request – same prefix, served from cache
ask("What was the gross margin in Q3?")

from openai import OpenAI

client = OpenAI(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a financial analyst assistant. You have access to the following
quarterly earnings report:

Fiscal Q3 2024 Earnings Report – Your Company

Revenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenue
was $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).

Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross margin
was 71.1%.

Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operating
income was $1.05 billion (25.0% margin).

Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net income
was $910 million, or $1.82 per diluted share.

Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.
Free cash flow was $980 million.

Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.

Replace this content with your actual document. The longer and more stable your system prompt,
the more tokens are eligible for caching."""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What was the revenue in Q3?")

# Second request – same prefix, served from cache
ask("What was the gross margin in Q3?")

Limitations

Prompt caching is available on MiniMax-M2.7 only. Other models return cached_tokens: 0.
Cache state is local to a single node and is not shared across nodes.
Maximum cacheable prefix length: 192000 tokens.
A prefix must contain at least 4096 tokens to qualify for caching.
Cache eviction uses an LRU (least recently used) policy. Cache persistence varies with system load and is not guaranteed.

​When to use prompt caching

​How it works

​Read cache usage from the response

​Billing

​Code example

​Limitations

When to use prompt caching

How it works

Read cache usage from the response

Billing

Code example

Limitations