Skip to main content
vLLM is an open source inference and serving engine for LLMs. vLLM serves an OpenAI-compatible API, so code written with the openai Python client for a self-hosted vLLM server can run against SambaCloud by changing the base URL, API key, and model. This page shows that change and what else to adjust.

Prerequisites

  • A SambaCloud account and API key
  • Python 3.10 or later and the openai package (pip install openai)

Point your OpenAI client at SambaCloud

vLLM’s own documentation shows querying a self-hosted server like this:
vllm_local.py
Pointed at SambaCloud instead, the key changes are base_url, api_key, and model:
sambanova_swap.py
Output (abridged)

vLLM-specific parameters

vLLM’s OpenAI-compatible server supports some parameters the standard OpenAI API doesn’t, like top_k, passed through the client’s extra_body argument:
SambaCloud accepts top_k through extra_body, the same way vLLM does. See OpenAI compatibility for the other parameters SambaCloud supports.

Tool calling

vLLM’s tool calling example uses the standard tools argument, including streaming. The non-streaming call works against SambaCloud with the same tools argument. The streaming loop needs one guard, described in the warning below:
Output
Every streamed response from SambaCloud ends with a final chunk whose choices list is empty (it only carries usage stats). vLLM’s own streaming example indexes chunk.choices[0] unconditionally, which raises IndexError on that last chunk. Guard for it: if not chunk.choices: continue before reading chunk.choices[0].delta.

Structured output (JSON schema)

vLLM’s structured outputs example uses response_format with a json_schema. SambaCloud accepts the same response_format:
Output

Troubleshooting

Export the key before running: export SAMBANOVA_API_KEY="your-key". Verify with echo $SAMBANOVA_API_KEY.
vLLM model names, such as NousResearch/Meta-Llama-3-8B-Instruct, are Hugging Face repo paths and don’t carry over. Use a SambaCloud model ID instead. See SambaCloud models for the current list.
The final chunk of every streamed response has an empty choices list (it only carries token usage). Skip chunks with no choices before indexing into chunk.choices[0], as described in Tool calling above.

Additional resources