Build with Vapi - SambaNova Integration Guide

Vapi is a developer platform for building voice AI agents. It handles the underlying infrastructure so developers can focus on creating voice experiences. Voice agents built with Vapi can:

Engage in natural conversations with users
Make and receive phone calls
Integrate with existing systems and APIs
Support complex workflows such as appointment scheduling, customer support, and other advanced use cases

SambaNova’s high-speed inference enables low-latency voice interactions, which is critical for natural-sounding conversations. This guide connects Vapi to SambaNova through a small Flask proxy that runs on your machine. Vapi calls the proxy as a Custom LLM provider, and the proxy forwards each request to SambaCloud. By the end you will have an assistant you can talk to in the browser.

Prerequisites

Before starting, ensure you have:

A SambaCloud account and API key (generate one from the API Keys tab in the SambaCloud portal).
A Vapi account with dashboard access. You do not need a phone number: Vapi can place a web call from the dashboard, so a working microphone and a browser that can access it are enough.
Python 3.9 or later.
ngrok installed, plus a free ngrok account. The account is required because ngrok will not open a tunnel until you add an auth token in Step 4.
Two terminals. The Flask server and the ngrok tunnel each run in the foreground and both must stay running for the rest of this guide.

macOS ships with Python 3.9 and exposes it as python3 rather than python. Check your version with python3 --version. Both dependencies used here declare requires-python >= 3.9, so the stock macOS interpreter is sufficient.

Choose a model

The proxy forwards the model name it receives straight through to SambaCloud, so the value you enter in Vapi must be an exact SambaCloud model ID. The examples in this guide use gpt-oss-120b.

Model ID	Context length	Developer Tier RPM	Developer Tier RPD
`gpt-oss-120b`	128k tokens	60	12,000
`Meta-Llama-3.3-70B-Instruct`	128k tokens	240	48,000

Use these facts to pick a model for voice:

Every conversational turn is one request. Vapi sends the whole transcript to your proxy each time the caller stops speaking, so a single call consumes one request per turn. Free Tier accounts are capped at 20 requests per day per model, which is roughly 20 turns before the agent starts failing mid-call. Link a payment method to move to the Developer Tier.
Meta-Llama-3.3-70B-Instruct has the most request headroom. At 240 RPM and 48,000 RPD on the Developer Tier it allows four times the request allowance of every other production model, which matters if you expect concurrent calls.
Keep streaming on. The proxy only streams when the request sets "stream": true, and Vapi’s speech synthesis begins from the first token it receives. Without streaming, the caller waits for the entire reply to finish generating before hearing anything.
Keep replies short. In voice, a long reply is a long stretch of the caller listening. app.py caps responses at 250 tokens when Vapi does not send its own max_tokens.

For the full model list and context lengths, see SambaCloud models. For the complete rate limit tables, see Rate limits.

Installation and setup

Follow the steps below to clone the repository, configure your environment, and connect SambaNova to Vapi through a local proxy server:

Step 1: Clone the repository

Clone the SambaNova integrations repository and navigate to the Vapi directory:

git clone https://github.com/sambanova/integrations.git
cd integrations/vapi

Step 2: Create a virtual environment

Create and activate a Python virtual environment to isolate your project dependencies:

python3 -m venv .venv
source .venv/bin/activate

Confirm the environment uses a supported version before installing:

python --version

Step 3: Install dependencies

Install the required Python libraries:

pip install "flask>=3.1,<4" "sambanova>=1.12,<2"

Step 4: Install ngrok

ngrok is required to expose your local server to the internet so that Vapi can reach it. On macOS, install ngrok using Homebrew:

brew install ngrok

On Linux and Windows, follow the ngrok installation guide for your platform. After installation, authenticate ngrok with your auth token. You can find your token in the ngrok dashboard. For more information, see the ngrok documentation:

ngrok config add-authtoken $YOUR_NGROK_AUTHTOKEN

Step 5: Set your API key

Export your SambaNova API key as an environment variable. Replace your-sambanova-api-key with the key from your SambaCloud portal:

export SAMBANOVA_API_KEY="your-sambanova-api-key"

The app.py in the integrations repository currently hard-codes a placeholder key and does not read this environment variable. Until that is fixed upstream, you must edit app.py yourself or every request fails with AuthenticationError: Error code: 401 - Incorrect API key provided: YOUR_S*****_KEY.

Open app.py and delete the hard-coded key. The SambaNova client reads SAMBANOVA_API_KEY from the environment on its own, so no other change is needed:

# Replace this line:
client = SambaNova(api_key="YOUR_SAMBANOVA_API_KEY")

# With this one:
client = SambaNova()

If the variable is not set, the client raises

SambaNovaError: The api_key client option must be set either by passing api_key to the client or by setting the SAMBANOVA_API_KEY environment variable

Step 6: Run the local LLM server

Start the Flask server in your first terminal, and leave it running:

python app.py

The server listens on port 5000 and exposes a single route, POST /chat/completions. Opening http://localhost:5000/ in a browser returns a 404, which is expected. Watch this terminal for the rest of the guide: every request Vapi makes appears here as a POST /chat/completions log line, which is the fastest way to confirm traffic is arriving.

Step 7: Expose the server using ngrok

In a separate terminal, run:

ngrok http 5000

ngrok will generate a public URL similar to:

https://abcd-1234.ngrok-free.dev

This is the endpoint Vapi will call.

app.py calls app.run(debug=True), which serves the interactive Werkzeug debugger, and the endpoint has no authentication. While the tunnel is open, anyone with the ngrok URL can send requests that consume your SambaNova quota, and any unhandled error returns a debugger page that exposes source code, local variables, and an evaluation console. Do not leave this tunnel running, do not share the URL, and never deploy this server as-is.To close the debugger exposure while you test, change the last line of app.py to app.run(port=5000).

Step 8: Test the endpoint

Verify your setup with a cURL request:

curl -X POST https://abcd-1234.ngrok-free.dev/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "call": "chat.completions",
    "metadata": {
      "request_id": "example-123"
    },
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello! Explain what an LLM is in one sentence."
      }
    ],
    "temperature": 0.7,
    "max_tokens": 150,
    "stream": true
  }'

Replace https://abcd-1234.ngrok-free.dev with your actual ngrok URL.

The request sets "stream": true, so the server responds with server-sent events rather than a single JSON body. app.py serializes each chunk and emits it as one data: line:

data: {"id":"...","object":"chat.completion.chunk","choices":[{"delta":{"content":"A"},...}]}

data: {"id":"...","object":"chat.completion.chunk","choices":[{"delta":{"content":" large"},...}]}

A stream of data: lines means the proxy is reaching SambaNova and Vapi will be able to consume it. An HTML page instead means the server raised an error; see Troubleshooting.

Step 9: Configure Vapi with your custom LLM

Log in to the Vapi Dashboard.
Select Create Assistant and start from a blank template.
In the assistant’s Model section, set the provider to Custom LLM.
Enter the model name. This value is forwarded to SambaCloud verbatim, so it must be an exact model ID from SambaCloud models (for example, gpt-oss-120b). A name that does not match exactly, including its capitalization, fails on the first turn.
Paste your ngrok URL into the endpoint URL field, without a path:
https://abcd-1234.ngrok-free.dev
Vapi uses this value as the OpenAI client’s base URL and appends /chat/completions itself, which matches the route the proxy server exposes. Including the path here produces /chat/completions/chat/completions and the request returns a 404.
Save the configuration.

ngrok issues a new public URL each time it restarts, which silently invalidates this setting. If Vapi stops reaching your server, check the current ngrok URL and update the endpoint.

Step 10: Talk to your assistant

With the Flask server and the ngrok tunnel both still running, select Talk in the Vapi dashboard. Vapi starts a web call in the browser, so no phone number is required. Grant microphone access when prompted, then speak to the assistant. You know the integration is working when all three of these happen:

Your Flask terminal logs a POST /chat/completions line with a 200 status for each thing you say.
The assistant replies out loud, and the reply begins while it is still being generated rather than arriving all at once.
The call appears in the Vapi dashboard with no LLM pipeline error.

ngrok runs a local request inspector at http://127.0.0.1:4040 while the tunnel is open. Open it in a browser to see the exact JSON body Vapi sent and the response your server returned, which is the quickest way to tell a Vapi configuration problem apart from a SambaNova one.

Troubleshooting

command not found: python

macOS does not provide a python executable, only python3. Use python3 -m venv .venv to create the environment. After you activate it with source .venv/bin/activate, python works as expected inside the environment.

AuthenticationError: Error code: 401 - Incorrect API key provided: YOUR_S*****_KEY

app.py is still using its placeholder key. Exporting SAMBANOVA_API_KEY alone has no effect, because the file does not read the environment. Apply the edit in Step 5, then restart the server.

OSError: [Errno 48] Address already in use

Another process holds port 5000. On macOS this is often AirPlay Receiver, which can occupy port 5000 and is turned on by default; you can disable it under System Settings → General → AirDrop & Handoff. Otherwise, move the server to a free port by changing the last line of app.py to app.run(port=5001), then run ngrok http 5001 in Step 7 instead.

Vapi returns 404 when it calls your endpoint

The endpoint URL in the Vapi dashboard includes a path. Vapi appends /chat/completions to whatever you enter, so a full path becomes /chat/completions/chat/completions, which the proxy server does not serve. Enter the ngrok base URL only, with no path.

Vapi stops reaching your server after it worked

ngrok issues a new public URL each time it restarts, and the old one stops resolving. The Vapi dashboard keeps the stale value with no warning. Check your current ngrok URL and update the endpoint in the assistant configuration.

The server returns an interactive debugger page instead of an error

app.py runs with debug=True, which serves the Werkzeug debugger on any unhandled exception. Over a public tunnel that exposes source, local variables, and an evaluation console. Change the last line of app.py to app.run(port=5000) to serve a plain error page instead.

The model name is rejected on the first turn

The name in the Vapi Model section is passed to SambaCloud unchanged, so anything that is not an exact model ID is rejected. Check the value against SambaCloud models, including its capitalization, and confirm it is not listed on the model deprecations page. Use the same ID in the Step 8 cURL request so both paths exercise the same model.

The call connects but the assistant never answers, or goes silent mid-conversation

Work outward from your machine. If your Flask terminal shows no POST /chat/completions line, Vapi never reached you: the tunnel is closed or the endpoint URL is stale, so recheck Step 7 and the URL saved in the dashboard. If the line is there but returned a 500, the proxy reached your server and failed downstream, most often on the API key from Step 5 or an invalid model name; the response body in the ngrok inspector at http://127.0.0.1:4040 names the cause. If the request succeeded and the caller still heard nothing, open the call in the Vapi dashboard and read its ended reason, documented in Vapi’s call ended reasons.

Requests start failing after the agent worked for a while

You are hitting a SambaCloud rate limit. Each conversational turn is one request, and Free Tier accounts are capped at 20 requests per day per model. Check the rate limit headers on the response and the tier tables on the Rate limits page, then link a payment method to move to the Developer Tier.

Additional resources

For detailed instructions and the complete source code, see the Vapi integration example on GitHub.
For more information about Vapi, see the official Vapi documentation.
For the models you can name in the Vapi dashboard, see SambaCloud models.
For per-model request and token limits, see Rate limits.
To call SambaNova directly instead of through a proxy, see the API reference.

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

Vapi integration guide

Prerequisites

Choose a model

Installation and setup

Step 1: Clone the repository

Step 2: Create a virtual environment

Step 3: Install dependencies

Step 4: Install ngrok

Step 5: Set your API key

Step 6: Run the local LLM server

Step 7: Expose the server using ngrok

Step 8: Test the endpoint

Step 9: Configure Vapi with your custom LLM

Step 10: Talk to your assistant

Troubleshooting

Additional resources

​Prerequisites

​Choose a model

​Installation and setup

​Step 1: Clone the repository

​Step 2: Create a virtual environment

​Step 3: Install dependencies

​Step 4: Install ngrok

​Step 5: Set your API key

​Step 6: Run the local LLM server

​Step 7: Expose the server using ngrok

​Step 8: Test the endpoint

​Step 9: Configure Vapi with your custom LLM

​Step 10: Talk to your assistant

​Troubleshooting

​Additional resources

Prerequisites

Choose a model

Installation and setup

Step 1: Clone the repository

Step 2: Create a virtual environment

Step 3: Install dependencies

Step 4: Install ngrok

Step 5: Set your API key

Step 6: Run the local LLM server

Step 7: Expose the server using ngrok

Step 8: Test the endpoint

Step 9: Configure Vapi with your custom LLM

Step 10: Talk to your assistant

Troubleshooting

Additional resources