openai Python client for a self-hosted vLLM server can run against SambaCloud by changing the base URL, API key, and model. This page shows that change and what else to adjust.
Prerequisites
- A SambaCloud account and API key
- Python 3.10 or later and the
openaipackage (pip install openai)
Point your OpenAI client at SambaCloud
vLLM’s own documentation shows querying a self-hosted server like this:vllm_local.py
base_url, api_key, and model:
sambanova_swap.py
Output (abridged)
vLLM-specific parameters
vLLM’s OpenAI-compatible server supports some parameters the standard OpenAI API doesn’t, liketop_k, passed through the client’s extra_body argument:
SambaCloud accepts
top_k through extra_body, the same way vLLM does. See OpenAI compatibility for the other parameters SambaCloud supports.Tool calling
vLLM’s tool calling example uses the standardtools argument, including streaming. The non-streaming call works against SambaCloud with the same tools argument. The streaming loop needs one guard, described in the warning below:
Output
Structured output (JSON schema)
vLLM’s structured outputs example usesresponse_format with a json_schema. SambaCloud accepts the same response_format:
Output
Troubleshooting
KeyError: SAMBANOVA_API_KEY
KeyError: SAMBANOVA_API_KEY
Export the key before running:
export SAMBANOVA_API_KEY="your-key". Verify with echo $SAMBANOVA_API_KEY.Model not found
Model not found
vLLM model names, such as
NousResearch/Meta-Llama-3-8B-Instruct, are Hugging Face repo paths and don’t carry over. Use a SambaCloud model ID instead. See SambaCloud models for the current list.IndexError: list index out of range while streaming
IndexError: list index out of range while streaming
The final chunk of every streamed response has an empty
choices list (it only carries token usage). Skip chunks with no choices before indexing into chunk.choices[0], as described in Tool calling above.
