Skip to main content
Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times.
Prompt caching is available on MiniMax-M2.7 on SambaCloud. Automatic Prefix Caching (APC) is enabled by default – no changes to your request are required.

​
When to use prompt caching

Prompt caching delivers the most savings when a large portion of your input is repeated across requests:
Good fitPoor fit
Long system prompt shared across all user turnsFully unique inputs per request (no shared prefix)
Repeated few-shot examples or tool definitionsVery short prompts where savings are negligible
RAG context re-sent with each querySingle-use documents where the prefix never repeats
Multi-turn conversations with a fixed preamble
If cached_tokens stays at 0 across requests, your inputs do not share a cacheable prefix – check whether your system prompt or context block is stable across calls.

​
How it works

Automatic Prefix Caching (APC) is always on for MiniMax-M2.7. It detects shared prefixes across requests and reuses cached computations transparently – no API changes required. Keep the leading portion of your messages stable and consistent across requests to maximize cache hits. A prefix is matched when the leading token sequence of the messages array is identical to a cached request. Any change to the prefix – rewording the system prompt, inserting a message before it, or reordering content – starts a new cache entry. Changing only the user message while keeping the system prompt identical reuses the cached prefix. Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.

​
Read cache usage from the response

Every response includes a prompt_tokens_details object showing how many tokens were served from cache: 
{
  "usage": {
    "prompt_tokens": 5797,
    "completion_tokens": 100,
    "total_tokens": 5897,
    "prompt_tokens_details": {
      "cache_creation_tokens": 0,
      "cached_tokens": 4096
    }
  }
}
FieldDescription
cached_tokensTokens served from cache. Billed at the cached input rate (lower than standard).
cache_creation_tokensTokens written to cache on this request. Informational only – no additional charge.
cache_creation_tokens is non-zero on the first request that builds the cache entry and 0 on subsequent cache hits. The number of non-cached input tokens is prompt_tokens - cached_tokens.

​
Billing

Cached tokens are billed at a lower rate than standard input tokens. The full pricing for your model – including the cached input rate and cache write rate – is available via the /v1/models endpoint and on the SambaCloud pricing page. 
curl https://cloud.sambanova.ai/v1/models \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  | jq '.data[] | select(.id == "MiniMax-M2.7") | .pricing'
The billing formula for a cache-enabled request: 
cost = (prompt_tokens − cached_tokens) × standard_input_rate
     + cached_tokens × cached_input_rate
     + completion_tokens × output_rate
For example, a request with 5797 total input tokens where 4096 are served from cache: only 1701 tokens are billed at the standard input rate, and 4096 at the lower cached rate. Output billing is unchanged.

​
Code example

Caching activates only when the shared prefix reaches 4096 tokens. The system prompt below is shorter for readability – replace it with your actual document to observe cache hits in practice.
from sambanova import SambaNova

client = SambaNova(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a financial analyst assistant. You have access to the following
quarterly earnings report:

Fiscal Q3 2024 Earnings Report – Your Company

Revenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenue
was $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).

Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross margin
was 71.1%.

Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operating
income was $1.05 billion (25.0% margin).

Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net income
was $910 million, or $1.82 per diluted share.

Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.
Free cash flow was $980 million.

Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.

Replace this content with your actual document. The longer and more stable your system prompt,
the more tokens are eligible for caching."""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What was the revenue in Q3?")

# Second request – same prefix, served from cache
ask("What was the gross margin in Q3?")

​
Limitations

  • Prompt caching is available on MiniMax-M2.7 only. Other models return cached_tokens: 0.
  • Cache state is local to a single node and is not shared across nodes.
  • Maximum cacheable prefix length: 192000 tokens.
  • A prefix must contain at least 4096 tokens to qualify for caching.
  • Cache eviction uses an LRU (least recently used) policy. Cache persistence varies with system load and is not guaranteed.