Use the SambaNova Cloud API with OpenAI’s client libraries to easily upgrade your existing applications to use the fastest inference on the best open source models!

​
Download the library

Run the command below to download the library.

pip install openai

​
Use SambaNova APIs with OpenAI client libraries

Switching to SambaNova APIs with OpenAI’s client libraries is as simple as setting two values, as shown below. Start a new file and create the client variable by passing the base_url and api_key. The api_key should be saved in a secure location. 

  1. Set the base_url to the “https://api.sambanova.ai/v1”.

  2. Set the api_key to the "YOUR SAMBANOVA CLOUD API KEY".

Don’t have a SambaNova Cloud API key? Get yours from the SambaNova Cloud portal.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.sambanova.ai/v1", 
    api_key="YOUR SAMBANOVA CLOUD API KEY"
)

Now you can make an API request to a model and choose how to receive your output. 

​
Non-streaming example

The following code demonstrates using the OpenAI python client for non-streaming completions.

completion = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages = [
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ]
)

print(completion.choices[0].message)

​
Streaming example

The following code demonstrates using the OpenAI python client for streaming completions.

completion = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages = [
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ],
  stream= True
)

for chunk in completion:
  print(chunk.choices[0].delta.content)

In streaming mode, the API returns chunks that contain multiple tokens. When calculating metrics like tokens per second or time per output token, ensure that you account for all tokens in each chunk.

​
Unsupported OpenAI features

The following features are not yet supported and will be ignored:

  • logprobs

  • top_logprobs

  • n

  • presence_penalty

  • frequency_penalty

  • logit_bias

  • seed

​
Feature differences

temperature: The SambaNova Cloud API supports a value between 0 and 1.

​
Features unsupported by OpenAI clients

The SambaNova API supports the top_k parameter. This is not supported by the OpenAI client.