> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM integration guide

[vLLM](https://docs.vllm.ai/) is an open source inference and serving engine for LLMs. vLLM serves an OpenAI-compatible API, so code written with the `openai` Python client for a self-hosted vLLM server can run against SambaCloud by changing the base URL, API key, and model. This page shows that change and what else to adjust.

## Prerequisites

* A [SambaCloud](https://cloud.sambanova.ai/?utm_source=vllm\&utm_medium=external\&utm_campaign=cloud_signup) account and [API key](https://cloud.sambanova.ai/apis?utm_source=vllm\&utm_medium=external\&utm_campaign=cloud_signup)
* Python 3.10 or later and the `openai` package (`pip install openai`)

## Point your OpenAI client at SambaCloud

vLLM's own documentation shows querying a self-hosted server like this:

```python vllm_local.py theme={}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message)
```

Pointed at SambaCloud instead, the key changes are `base_url`, `api_key`, and `model`:

```python sambanova_swap.py theme={}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.sambanova.ai/v1",
    api_key=os.environ["SAMBANOVA_API_KEY"],
)

completion = client.chat.completions.create(
    model="MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "What can you do?"}],
)
print(completion.choices[0].message.content)
```

```text Output (abridged) theme={}
I can help you think, write, learn, and solve problems. For example:

- **Answer questions** and explain complex topics in simple terms
- **Write and edit** emails, essays, stories, résumés, and reports
- **Write or debug code** and explain how software works
...
```

## vLLM-specific parameters

vLLM's OpenAI-compatible server supports some parameters the standard OpenAI API doesn't, like `top_k`, passed through the client's `extra_body` argument:

```python theme={}
completion = client.chat.completions.create(
    model="MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "What can you do?"}],
    extra_body={"top_k": 20},
)
```

> [!NOTE]
>
> SambaCloud accepts `top_k` through `extra_body`, the same way vLLM does. See [OpenAI compatibility](https://docs.sambanova.ai/docs/en/features/openai-compatibility.md) for the other parameters SambaCloud supports.

## Tool calling

vLLM's [tool calling example](https://github.com/vllm-project/vllm/blob/main/examples/tool_calling/openai_chat_completion_client_with_tools.py) uses the standard `tools` argument, including streaming. The non-streaming call works against SambaCloud with the same `tools` argument. The streaming loop needs one guard, described in the warning below:

```python theme={}
tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "state": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city", "state", "unit"],
        },
    },
}]

completion = client.chat.completions.create(
    model="MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "What's the temperature in Dallas, in fahrenheit?"}],
    tools=tools,  # same OpenAI "function" tool schema vLLM's example uses
)
print(completion.choices[0].message.tool_calls)
```

```text Output theme={}
[ChatCompletionMessageFunctionToolCall(id='call_4255df7f20184da18e', function=Function(arguments='{"city": "Dallas", "state": "TX", "unit": "fahrenheit"}', name='get_current_weather'), type='function')]
```

> [!WARNING]
>
> Every streamed response from SambaCloud ends with a final chunk whose `choices` list is empty (it only carries `usage` stats). vLLM's own streaming example indexes `chunk.choices[0]` unconditionally, which raises `IndexError` on that last chunk. Guard for it: `if not chunk.choices: continue` before reading `chunk.choices[0].delta`.

## Structured output (JSON schema)

vLLM's [structured outputs example](https://docs.vllm.ai/en/latest/features/structured_outputs.html) uses `response_format` with a `json_schema`. SambaCloud accepts the same `response_format`:

```python theme={}
from pydantic import BaseModel

class CarDescription(BaseModel):
    brand: str
    model: str
    car_type: str

completion = client.chat.completions.create(
    model="MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "car-description", "schema": CarDescription.model_json_schema()},
    },
)
print(completion.choices[0].message.content)
```

```text Output theme={}
{
  "brand": "Toyota",
  "car_type": "Sports car",
  "model": "Supra Mk4 (A80)"
}
```

## Troubleshooting

> [accordion] KeyError: SAMBANOVA\_API\_KEY
>
> Export the key before running: `export SAMBANOVA_API_KEY="your-key"`. Verify with `echo $SAMBANOVA_API_KEY`.

> [accordion] Model not found
>
> vLLM model names, such as `NousResearch/Meta-Llama-3-8B-Instruct`, are Hugging Face repo paths and don't carry over. Use a SambaCloud model ID instead. See [SambaCloud models](https://docs.sambanova.ai/docs/en/models/sambacloud-models.md) for the current list.

> [accordion] IndexError: list index out of range while streaming
>
> The final chunk of every streamed response has an empty `choices` list (it only carries token usage). Skip chunks with no choices before indexing into `chunk.choices[0]`, as described in [Tool calling](#tool-calling) above.

## Additional resources

* [vLLM documentation](https://docs.vllm.ai/)
* [vLLM OpenAI-compatible server](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server.html)
* [vLLM examples](https://docs.vllm.ai/en/latest/examples/)
* [vLLM tool calling example](https://github.com/vllm-project/vllm/blob/main/examples/tool_calling/openai_chat_completion_client_with_tools.py)
* [vLLM structured outputs](https://docs.vllm.ai/en/latest/features/structured_outputs.html)
* [Supported SambaCloud models](https://docs.sambanova.ai/docs/en/models/sambacloud-models.md)
