Build with pgvector - SambaNova Integration Guide

pgvector is a PostgreSQL extension for storing and searching vectors. You can use it with SambaNova by generating embeddings with a SambaStack embedding model, writing the vectors to a Postgres table, and answering questions with a SambaNova chat model.

Embeddings are available on SambaStack only. E5-Mistral-7B-Instruct is not available on SambaCloud. See Implement embeddings features.On SambaCloud, you can still use pgvector. Set SAMBANOVA_BASE_URL to https://api.sambanova.ai/v1, install FastEmbed with pip install fastembed, and replace the embed function in step 1 with one that embeds locally:

from fastembed import TextEmbedding

local_model = TextEmbedding()

def embed(texts):
    return [vector.tolist() for vector in local_model.embed(texts)]

The steps in Embed, store, and query work unchanged. These vectors have 384 dimensions, so you do not need Index 4096-dimension vectors. A standard hnsw index works.

How pgvector uses SambaNova

pgvector never calls an embedding or chat API. It stores vectors and compares them, so there is no model setting to configure. Your code makes both calls: it embeds text with SambaStack, writes the vectors to Postgres, and sends the retrieved text to a chat model. The same steps work on any Postgres that has pgvector, including managed services that offer it. E5-Mistral-7B-Instruct returns 4096-dimension vectors, which is more than pgvector can index with its standard options. This guide shows what works: exact search for small tables, and a binary quantization index for larger ones. See Index 4096-dimension vectors.

Prerequisites

A SambaStack instance that serves E5-Mistral-7B-Instruct, plus the instance’s base URL and API key. See API keys and URLs.
A chat model on that instance. This guide uses gpt-oss-120b.
A PostgreSQL database with pgvector 0.7.0 or higher, and a connection URL for it. Version 0.7.0 added the binary quantization function used below. To install pgvector, follow its installation instructions.
Python 3.10 or higher.

macOS ships with Python 3.9 and exposes it as python3 rather than python. Check your version with python3 --version. If it is below 3.10, install a supported version with pyenv, uv, or brew install python@3.12 before continuing.

Installation and setup

Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

python -m venv .venv
.venv\Scripts\Activate.ps1

Install dependencies

pip install "psycopg[binary]" openai

This guide was tested with psycopg 3.3.6, openai 3.35.1, PostgreSQL 17.11, and pgvector 0.8.7.

Set your SambaStack base URL, API key, and database URL

The SambaStack base URL must end in /v1.

export SAMBANOVA_BASE_URL="your-sambastack-base-url/v1"
export SAMBANOVA_API_KEY="your-sambanova-api-key"
export DATABASE_URL="postgresql://user:password@localhost:5432/your-database"

$env:SAMBANOVA_BASE_URL="your-sambastack-base-url/v1"
$env:SAMBANOVA_API_KEY="your-sambanova-api-key"
$env:DATABASE_URL="postgresql://user:password@localhost:5432/your-database"

Embed, store, and query

Run the steps in this section in one Python file or session. Each step uses variables from the steps before it.

Embed your documents

The OpenAI client works against the SambaStack embeddings endpoint.

import os
from openai import OpenAI

sn = OpenAI(
    base_url=os.environ["SAMBANOVA_BASE_URL"],
    api_key=os.environ["SAMBANOVA_API_KEY"],
)

docs = [
    "SambaNova builds the SN40L chip, a reconfigurable dataflow unit for fast AI inference.",
    "PostgreSQL is an open-source relational database, and pgvector adds vector search to it.",
    "Sourdough bread needs a live starter, flour, water, and salt, and a long slow fermentation.",
    "The capital of Brazil is Brasilia, which was inaugurated in 1960.",
]

def embed(texts):
    response = sn.embeddings.create(model="E5-Mistral-7B-Instruct", input=texts)
    return [d.embedding for d in response.data]

vectors = embed(docs)
dimension = len(vectors[0])  # 4096

The embeddings endpoint accepts a dimensions argument but ignores it, so you cannot shorten the vectors this way. Every vector has 4096 values.

Create a table and insert the vectors

Enable the extension, then create a vector column whose size matches dimension. A vector column holds up to 16,000 dimensions. The DROP TABLE IF EXISTS line lets you run the guide again without an already exists error.

import psycopg

conn = psycopg.connect(os.environ["DATABASE_URL"], autocommit=True)
conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
conn.execute("DROP TABLE IF EXISTS items")
conn.execute(f"CREATE TABLE items (id bigserial PRIMARY KEY, content text, embedding vector({dimension}))")

for content, vector in zip(docs, vectors):
    conn.execute(
        "INSERT INTO items (content, embedding) VALUES (%s, %s::vector)",
        (content, vector),
    )

Cast every Python list to vector with ::vector. Without the cast, Postgres sees the list as a double precision[] and the distance operators fail.

Query the table

E5-Mistral-7B-Instruct is an instruction-tuned model, so write each query as an instruction followed by the question. Documents are embedded as plain text. The <=> operator returns cosine distance, so 1 - distance is the cosine similarity.

task = "Given a question, retrieve passages that answer the question"
question = "Which chip does SambaNova make?"

query_vector = embed([f"Instruct: {task}\nQuery: {question}"])[0]
rows = conn.execute(
    """
    SELECT content, 1 - (embedding <=> %s::vector) AS score
    FROM items
    ORDER BY embedding <=> %s::vector
    LIMIT 2
    """,
    (query_vector, query_vector),
).fetchall()

for content, score in rows:
    print(f"{score:.3f} {content}")

The closest match is the passage about the SN40L chip, with a cosine score of about 0.71. The next closest scores about 0.52. Higher is closer.Without an index, Postgres compares the query with every row, so the results are exact.

Answer with a SambaNova chat model

Pass the retrieved passages to a chat model as context.

context = "\n".join(content for content, _ in rows)

answer = sn.chat.completions.create(
    model="gpt-oss-120b",
    messages=[
        {"role": "system", "content": f"Answer using only the context below.\n\nContext:\n{context}"},
        {"role": "user", "content": question},
    ],
)
print(answer.choices[0].message.content)

question is the same question you searched with in step 3.The model answers that SambaNova makes the SN40L chip.

Index 4096-dimension vectors

For large tables, an approximate index keeps queries fast. pgvector’s hnsw and ivfflat indexes cannot cover 4096 dimensions. Their limits are 2,000 dimensions for vector and 4,000 for halfvec, so both of these fail:

CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
-- ERROR: column cannot have more than 2000 dimensions for hnsw index

CREATE INDEX ON items USING hnsw ((embedding::halfvec(4096)) halfvec_cosine_ops);
-- ERROR: column cannot have more than 4000 dimensions for hnsw index

Binary quantization indexes up to 64,000 dimensions. It keeps one bit per dimension in the index, finds a shortlist of candidates by Hamming distance, and then reranks the shortlist with exact cosine distance. Create the index on an expression, so the table keeps the full vectors:

conn.execute(
    f"""
    CREATE INDEX items_embedding_bq ON items
    USING hnsw ((binary_quantize(embedding)::bit({dimension})) bit_hamming_ops)
    """
)

Query it by shortlisting with the quantized distance, then reranking:

rows = conn.execute(
    f"""
    SELECT content, 1 - (embedding <=> %s::vector) AS score
    FROM (
        SELECT * FROM items
        ORDER BY binary_quantize(embedding)::bit({dimension}) <~> binary_quantize(%s::vector)
        LIMIT 20
    ) AS shortlist
    ORDER BY embedding <=> %s::vector
    LIMIT 2
    """,
    (query_vector, query_vector, query_vector),
).fetchall()

for content, score in rows:
    print(f"{score:.3f} {content}")

The shortlist size (LIMIT 20) trades speed for recall. A larger shortlist reranks more candidates. On this four-row table the results match the exact query. The index expression in the query must match the one in CREATE INDEX exactly, or Postgres will not use the index. To check, run EXPLAIN on the shortlist query and look for Index Scan using items_embedding_bq. On a table this small, Postgres may choose a sequential scan. Run SET enable_seqscan = off first to confirm the index can be used.

Clean up

conn.execute("DROP TABLE items")
conn.close()

Additional resources

pgvector on GitHub

Installation, operators, index types, and limits.

psycopg documentation

The Python driver used in this guide to connect to PostgreSQL.

FastEmbed

Local embedding library, used in the SambaCloud tip above.

Implement embeddings features

How to use SambaNova embedding models on SambaStack.

Troubleshooting

KeyError: 'DATABASE_URL'

An environment variable from the setup step is not set in the shell you run Python from. The error names the missing variable, such as DATABASE_URL, SAMBANOVA_BASE_URL, or SAMBANOVA_API_KEY. Set it as shown in Installation and setup, then run the code again from the same shell.

column cannot have more than 2000 dimensions for hnsw index

The same error ends in for ivfflat index when you create an ivfflat index. A vector column with 4096 dimensions is too large for an hnsw or ivfflat index. Use the binary quantization index in Index 4096-dimension vectors, or search without an index.

column cannot have more than 4000 dimensions for hnsw index

Casting to halfvec does not help, because its limit is 4,000 dimensions and the model returns 4096. Use binary quantization instead.

operator does not exist: vector <=> double precision[]

The query vector reached Postgres as a plain array. Cast it with %s::vector, as in the examples above.

function binary_quantize(vector) does not exist

Your pgvector is older than 0.7.0. Upgrade pgvector, then run ALTER EXTENSION vector UPDATE in the database.

401 invalid_api_key from the embeddings endpoint

The SambaNova key is wrong for the base URL. Use the key issued for your SambaStack instance, and make sure SAMBANOVA_BASE_URL points at that instance and ends in /v1. Without /v1, the embeddings request returns 404 page not found.

404 model_not_found for E5-Mistral-7B-Instruct

The instance you are calling does not serve the model, or your key cannot access it. List the models the instance serves:

curl -H "Authorization: Bearer $SAMBANOVA_API_KEY" "$SAMBANOVA_BASE_URL/models"

curl.exe -H "Authorization: Bearer $env:SAMBANOVA_API_KEY" "$env:SAMBANOVA_BASE_URL/models"

SambaCloud does not serve embeddings. See the SambaCloud tip at the top of this page.

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

pgvector integration guide

How pgvector uses SambaNova

Prerequisites

Installation and setup

Embed, store, and query

Index 4096-dimension vectors

Clean up

Additional resources

pgvector on GitHub

psycopg documentation

FastEmbed

Implement embeddings features

Troubleshooting

​How pgvector uses SambaNova

​Prerequisites

​Installation and setup

​Embed, store, and query

​Index 4096-dimension vectors

​Clean up

​Additional resources

pgvector on GitHub

psycopg documentation

FastEmbed

Implement embeddings features

​Troubleshooting

How pgvector uses SambaNova

Prerequisites

Installation and setup

Embed, store, and query

Index 4096-dimension vectors

Clean up

Additional resources

Troubleshooting