> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Integration with LiteLLM

<Note>
  LiteLLM is a third-party proxy service. SambaNova doesn't endorse, maintain, or audit LiteLLM's security or functionality. This guide is provided for informational purposes and may become outdated. Use at your own discretion.
</Note>

LiteLLM provides a gateway layer that sits in front of SambaStack's OpenAI-compatible inference APIs, adding enterprise-grade controls.

| Capability                 | Description                                          |
| :------------------------- | :--------------------------------------------------- |
| Centralized authentication | Single point for API key management and rotation     |
| Usage tracking             | Monitor usage across teams, projects, and keys       |
| Cost controls              | Implement budgets, quotas, and rate limits           |
| Audit logging              | Record model interactions for compliance and reviews |
| Model routing              | Switch deployments/providers without code changes    |

<Info>
  **Target audience:** Engineers integrating a control layer (rate limits, key management, routing) on top of SambaStack's OpenAI-compatible inference APIs.
</Info>

## Architecture

SambaStack runs your model deployments on dedicated RDU nodes and exposes them behind an OpenAI-compatible HTTPS API. LiteLLM sits in front of those endpoints as a proxy gateway.

<Frame>
  <img src="https://mintcdn.com/sambanova-systems/emEy6D0rW6i9bD6c/images/docs/admin/litellm/litellm-arch.png?fit=max&auto=format&n=emEy6D0rW6i9bD6c&q=85&s=e32ab6c65619ab7b540f66f3c38da00a" style={{ maxWidth: "120%" }} alt="LiteLLM architecture diagram showing LiteLLM proxy between client applications and SambaStack inference API" width="1024" height="189" data-path="images/docs/admin/litellm/litellm-arch.png" />
</Frame>

<Accordion title="Responsibility matrix">
  | Layer                    | SambaStack                                                   | LiteLLM                                                 | Your App                                 |
  | :----------------------- | :----------------------------------------------------------- | :------------------------------------------------------ | :--------------------------------------- |
  | **Hardware & runtime**   | Runs models on RDU nodes; manages kernels, batching, queuing | –                                                       | –                                        |
  | **Inference API**        | Provides OpenAI-compatible endpoints                         | Proxies endpoints behind a single gateway URL           | Calls SambaStack directly or via LiteLLM |
  | **Traffic controls**     | Platform-level limits                                        | Per-key, user, model, team rate limits, quotas, budgets | Optional safeguard logic                 |
  | **API key management**   | Issues/validates platform API keys                           | Rotate, scope, revoke app keys; map app→platform keys   | Store app-level secrets                  |
  | **Model access control** | Platform API keys grant access to deployments                | Per-app-key allow/deny by model alias                   | Select allowed aliases per role/policy   |
</Accordion>

## Prerequisites

Before starting, ensure you have:

* Python 3.11 or above with pip
* PostgreSQL database (local, self-hosted, or managed)
* SambaStack API key and base URL

You'll also need to configure these environment variables:

| Variable              | Description                                                                            |
| :-------------------- | :------------------------------------------------------------------------------------- |
| `SAMBASTACK_BASE_URL` | Base URL for SambaStack's OpenAI-compatible API                                        |
| `SAMBASTACK_API_KEY`  | Platform API key used by LiteLLM to call SambaStack                                    |
| `DATABASE_URL`        | PostgreSQL connection string (format: `postgresql://user:password@host:port/database`) |
| `LITELLM_MASTER_KEY`  | Admin key for the LiteLLM proxy and API                                                |
| `LITELLM_SALT_KEY`    | Salt for encrypting credentials in DB (cannot be changed once set)                     |
| `PORT`                | Port for the LiteLLM HTTP server (default: `4000`)                                     |
| `STORE_MODEL_IN_DB`   | Set to `True` to store model definitions in DB                                         |

## Getting started

For additional installation options, refer to the [official LiteLLM documentation](https://docs.litellm.ai/).

### Step 1: Set up virtual environment

```bash theme={null}
python3 -m venv litellm-env
source litellm-env/bin/activate
pip install "litellm[proxy]" prisma
```

### Step 2: Configure environment variables

```bash theme={null}
export SAMBASTACK_BASE_URL="https://your-sambastack-instance.example.com/v1"
export SAMBASTACK_API_KEY="your-sambastack-api-key"
export DATABASE_URL="postgresql://litellm_admin:YOUR_PASSWORD@localhost:5432/sambastack_litellm"
export LITELLM_MASTER_KEY="sk-admin-your-master-key"
export LITELLM_SALT_KEY="sk-salt-your-salt-key"
export PORT=4000
export STORE_MODEL_IN_DB=True
```

### Step 3: Set up database

Connect to PostgreSQL as a superuser:

```bash theme={null}
psql -U <SUPERUSER> -d postgres
```

Create the database and user:

```sql theme={null}
CREATE USER litellm_admin WITH PASSWORD 'YOUR_PASSWORD';
CREATE DATABASE sambastack_litellm OWNER litellm_admin;
GRANT ALL PRIVILEGES ON DATABASE sambastack_litellm TO litellm_admin;
\c sambastack_litellm
ALTER SCHEMA public OWNER TO litellm_admin;
```

### Step 4: Initialize database schema

```bash theme={null}
# Find the schema path dynamically
SCHEMA_PATH=$(python -c "import litellm_proxy_extras; print(litellm_proxy_extras.__path__[0])")/schema.prisma

# Generate client and create tables
prisma generate --schema "$SCHEMA_PATH"
prisma db push --schema "$SCHEMA_PATH"
```

### Step 5: Create configuration file

Create `litellm_config.yaml`:

```yaml theme={null}
model_list:
  - model_name: "<MODEL_NAME_1>"
    litellm_params:
      model: "sambanova/<MODEL_NAME_1>"
      api_base: "os.environ/SAMBASTACK_BASE_URL"
      api_key: "os.environ/SAMBASTACK_API_KEY"
      rpm: 5
  - model_name: "<MODEL_NAME_2>"
    litellm_params:
      model: "sambanova/<MODEL_NAME_2>"
      api_base: "os.environ/SAMBASTACK_BASE_URL"
      api_key: "os.environ/SAMBASTACK_API_KEY"
      rpm: 5

general_settings:
  database_url: "os.environ/DATABASE_URL"
  database_connection_pool_limit: 100
  database_connection_timeout: 60
```

<Warning>
  Replace `<MODEL_NAME_1>` and `<MODEL_NAME_2>` with your actual model names from SambaStack.
</Warning>

### Step 6: Run LiteLLM proxy

```bash theme={null}
litellm --config ./litellm_config.yaml --detailed_debug
```

### Step 7: Verify installation

<Steps>
  <Step title="Login to LiteLLM UI">
    Navigate to `http://localhost:4000/`. Use `admin` as the username and your master key as the password.
  </Step>

  <Step title="Verify model connection">
    Go to **Model Management → Health Status**. Your model should appear as connected to SambaStack.
  </Step>

  <Step title="Run health check">
    Click Health Check to confirm connectivity.

    <img src="https://mintcdn.com/sambanova-systems/emEy6D0rW6i9bD6c/images/docs/admin/litellm/litellm-health-check.png?fit=max&auto=format&n=emEy6D0rW6i9bD6c&q=85&s=65fa8522f1d2d8212c1102042243bba7" alt="LiteLLM Health Check" width="3516" height="1966" data-path="images/docs/admin/litellm/litellm-health-check.png" />
  </Step>
</Steps>

## Teams, keys, and rate limits

For the full feature set, refer to the [official LiteLLM documentation](https://docs.litellm.ai/).

### Create a team

<Steps>
  <Step title="Navigate to Teams">
    Go to **Teams → Create New Team**
  </Step>

  <Step title="Configure team settings">
    * **Team name:** e.g., `Test-team`
    * **Models:** Select your SambaStack model
    * **Max Budget (USD):** e.g., `10`
    * **Reset Budget:** e.g., `monthly`
    * **RPM limit:** e.g., `3`

          <img src="https://mintcdn.com/sambanova-systems/emEy6D0rW6i9bD6c/images/docs/admin/litellm/litellm-team-creation.png?fit=max&auto=format&n=emEy6D0rW6i9bD6c&q=85&s=ae7ddd0471dbb86c51682dfd6b8fb92f" alt="LiteLLM Team Creation" width="3510" height="1954" data-path="images/docs/admin/litellm/litellm-team-creation.png" />
  </Step>

  <Step title="Save">
    Click **Save** to create the team.
  </Step>
</Steps>

### Invite a user

<Steps>
  <Step title="Navigate to Internal Users">
    Go to **Internal Users → Invite User**
  </Step>

  <Step title="Configure user">
    Enter email, choose a role, and assign to your team.

    <img src="https://mintcdn.com/sambanova-systems/emEy6D0rW6i9bD6c/images/docs/admin/litellm/litellm-invite-user.png?fit=max&auto=format&n=emEy6D0rW6i9bD6c&q=85&s=1198674ec446e157e9eda1a85adab9a5" alt="LiteLLM Invite User" width="3520" height="1956" data-path="images/docs/admin/litellm/litellm-invite-user.png" />
  </Step>

  <Step title="Send invitation">
    Copy the invitation link and share it with the user.
  </Step>
</Steps>

### Create virtual API keys

<Steps>
  <Step title="Navigate to Virtual Keys">
    Go to **Virtual Keys → Create New Key**
  </Step>

  <Step title="Configure key">
    * **Team:** Select your team
    * **Key Name:** e.g., `limited-key`
    * **Models:** Choose All Team Models or specific model

          <img src="https://mintcdn.com/sambanova-systems/emEy6D0rW6i9bD6c/images/docs/admin/litellm/litellm-key-creation.png?fit=max&auto=format&n=emEy6D0rW6i9bD6c&q=85&s=bbd2bb77e78cd471fa019e78aefd9651" alt="LiteLLM Key Creation" width="3526" height="1868" data-path="images/docs/admin/litellm/litellm-key-creation.png" />
  </Step>

  <Step title="Create and copy key">
    Click **Create Key** and copy immediately. This is shown only once.
  </Step>
</Steps>

### Test the virtual API key

```bash theme={null}
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR_VIRTUAL_KEY>" \
  -d '{
    "model": "<MODEL_NAME>",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

### Monitor usage and logs

* **Usage:** View spend, requests, and tokens by time range and team at **Usage** tab
* **Logs:** Review request outcomes including rate-limit failures at **Logs** tab

<Note>
  By default, request/response bodies aren't stored. Enable storage in the proxy configuration if required for compliance.
</Note>

## In-cluster installation

For production, install LiteLLM in the same Kubernetes cluster as SambaStack for optimal performance. See the [LiteLLM Kubernetes deployment guide](https://docs.litellm.ai/docs/proxy/deploy).

## Troubleshooting

### Health check fails

**Symptom:** Model health check returns connection error.

**Solutions:**

* Verify `SAMBASTACK_BASE_URL` is correct and accessible
* Confirm `SAMBASTACK_API_KEY` is valid
* Check network connectivity between LiteLLM and SambaStack

### Database connection errors

**Symptom:** LiteLLM fails to start with database errors.

**Solutions:**

* Verify `DATABASE_URL` format is correct
* Confirm PostgreSQL is running and accessible
* Check that database and user exist with correct permissions

### Prisma schema not found

**Symptom:** `prisma generate` fails with "schema not found" error.

**Solution:** Ensure virtual environment is activated and `litellm[proxy]` is installed. Use the dynamic path detection shown in Step 4.

## Additional resources

* [LiteLLM Documentation](https://docs.litellm.ai/)
* [LiteLLM Kubernetes Deployment](https://docs.litellm.ai/docs/proxy/deploy)
* [LiteLLM API Reference](https://docs.litellm.ai/docs/proxy/api_reference)