> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Using prompt caching

Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times.

Prompt caching is available for **MiniMax-M2.7** on SambaStack 1.2.

<Info>
  This page covers how to use prompt caching and interpret usage fields. For model availability, see [Supported models and bundles](/en/v1.2.0/sambastack/service-administration/supported-models-and-bundles).
</Info>

## When to use prompt caching

Prompt caching delivers the most savings when a large portion of your input is repeated across requests:

| Good fit                                        | Poor fit                                                    |
| ----------------------------------------------- | ----------------------------------------------------------- |
| Long system prompt shared across all user turns | Fully unique inputs per request (no shared prefix)          |
| Repeated few-shot examples or tool definitions  | Very short prompts where caching overhead outweighs savings |
| RAG context re-sent with each query             | Single-use documents where the prefix never repeats         |
| Multi-turn conversations with a fixed preamble  |                                                             |

If `cached_tokens` stays at `0` across requests, your inputs do not share a cacheable prefix – review whether your system prompt or context block is stable across calls.

## How prompt caching works

**Automatic Prefix Caching (APC)** is available for MiniMax-M2.7 but must be enabled by a SambaStack administrator by deploying the `dyt-minimax-m2p7-32-64-192k-pc` bundle. Once enabled, APC detects shared prefixes across requests and reuses cached computations transparently – no changes to your API requests are required.

Prompt caching follows the OpenAI standard. Cache hits are detected automatically and reported in the response `usage` field – no `cache_control` markers or other request body changes are needed.

APC is local to each serving instance. Cached KV state is stored in local DDR and is not transferred across instances. On multi-instance deployments, your prefix is cached independently on each instance after it is served there. The sequence state pool size for MiniMax-M2.7 is 192K.

## Checking cache usage in the response

Every `/v1/chat/completions` response from a cache-enabled model includes cache token counts in the `usage` field. The following example shows a second request that shares the same long system prompt as a prior request – the prefix is served from cache rather than reprocessed.

**Request (second call – cache warm)**

```bash theme={null}
# A long, repeated system prompt (1,000+ tokens) triggers APC on the second call.
curl https://<sambastack-host>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMax-M2.7",
    "messages": [
      {
        "role": "system",
        "content": "You are a technical assistant for SambaNova hardware. ..."
      },
      {
        "role": "user",
        "content": "What is the default batch size for MiniMax-M2.7?"
      }
    ],
    "max_tokens": 100
  }'
```

**Response**

```json theme={null}
{
  "id": "cmpl-3ca55e7f-8fd5-470d-bac6-ee38a7a5307b",
  "object": "chat.completion",
  "created": 1779835973,
  "model": "MiniMax-M2.7",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The default batch size for MiniMax-M2.7 is 2."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5797,
    "completion_tokens": 100,
    "total_tokens": 5897,
    "prompt_tokens_details": {
      "cache_creation_tokens": 0,
      "cached_tokens": 4096
    }
  }
}
```

| Field                   | Description                                                                                         |
| ----------------------- | --------------------------------------------------------------------------------------------------- |
| `cached_tokens`         | Input tokens served from the cache. These are billed at the cached input rate.                      |
| `cache_creation_tokens` | Input tokens used to populate the cache on this request. Informational only – no additional charge. |

**Input remaining** = `prompt_tokens` − `cached_tokens`. This portion is billed at the standard input rate.

A `cached_tokens` value of `0` means no cache hit occurred on that request – it is not an error. This happens on the first request for a given prefix (cache is being populated) or when the input does not share a prefix with any cached entry.

For models that do not support prompt caching, `cached_tokens` is always `0` and billing is unchanged.

## Code example

<CodeGroup>
  ```python Python (SambaNova) theme={null}
  from sambanova import SambaNova

  client = SambaNova(
      base_url="https://<sambastack-host>",
      api_key="your-sambanova-api-key",
  )

  system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
  the following reference documentation:

  [Insert your long, stable system prompt or document here – the longer and more stable this
  prefix, the more tokens are eligible for caching. A prefix must be at least 4,096 tokens
  to qualify.]"""

  def ask(question):
      response = client.chat.completions.create(
          model="MiniMax-M2.7",
          messages=[
              {"role": "system", "content": system_prompt},
              {"role": "user", "content": question},
          ],
      )
      usage = response.usage
      cached = usage.prompt_tokens_details.cached_tokens
      total_prompt = usage.prompt_tokens
      print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
      print(response.choices[0].message.content)

  # First request – populates the cache
  ask("What is the default batch size for MiniMax-M2.7?")

  # Second request – same prefix, served from cache
  ask("What is the maximum sequence length for MiniMax-M2.7?")
  ```

  ```python Python (OpenAI) theme={null}
  from openai import OpenAI

  client = OpenAI(
      base_url="https://<sambastack-host>/v1",
      api_key="your-sambanova-api-key",
  )

  system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
  the following reference documentation:

  [Insert your long, stable system prompt or document here – the longer and more stable this
  prefix, the more tokens are eligible for caching. A prefix must be at least 4,096 tokens
  to qualify.]"""

  def ask(question):
      response = client.chat.completions.create(
          model="MiniMax-M2.7",
          messages=[
              {"role": "system", "content": system_prompt},
              {"role": "user", "content": question},
          ],
      )
      usage = response.usage
      cached = usage.prompt_tokens_details.cached_tokens
      total_prompt = usage.prompt_tokens
      print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
      print(response.choices[0].message.content)

  # First request – populates the cache
  ask("What is the default batch size for MiniMax-M2.7?")

  # Second request – same prefix, served from cache
  ask("What is the maximum sequence length for MiniMax-M2.7?")
  ```
</CodeGroup>

## Billing impact

Cached tokens are billed at a lower rate than standard input tokens. The billing calculation for a cache-enabled request is:

| Token type                                          | Billing                          |
| --------------------------------------------------- | -------------------------------- |
| Input remaining (`prompt_tokens` − `cached_tokens`) | Standard input rate              |
| `cached_tokens`                                     | Cached input rate (lower)        |
| `completion_tokens`                                 | Standard output rate (unchanged) |

For example, a request with 5,797 total input tokens where 4,096 are served from cache: you pay the standard input rate on 1,701 tokens and the cached input rate on 4,096 tokens. Output billing is unchanged.

`cache_creation_tokens` appears in the response but is informational only – it represents tokens used to populate the cache on this request and carries no additional charge at this time.

For models without prompt caching, `cached_tokens` is always `0`, so the formula reduces to the standard calculation with no billing change.

<Note>
  Contact your SambaNova representative for current pricing, including the cached input rate for MiniMax-M2.7.
</Note>

## Limitations

* Prompt caching is available on **MiniMax-M2.7** only in SambaStack 1.2.
* Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.
* APC operates on shared prefixes. Requests that diverge before the cached boundary do not get a cache hit.
* The maximum cached prefix (sequence state) is **192K tokens**.
* Cache eviction uses an **LRU (least recently used)** policy. Cache persistence duration varies with system load and is not guaranteed.
* A prefix must contain at least **4,096 tokens** to qualify for caching.