Build with Ragas - SambaNova Integration Guide

Ragas is a metric library for evaluating RAG pipelines. Judge-model calls are the core of how it scores responses, so a fast judge directly speeds up every evaluation run.

Prerequisites

A SambaCloud account and API key
Python 3.9 or later

Setup

Install Ragas

pip install "ragas==0.4.3" openai
pip install "langchain-community==0.3.30" --no-deps

Ragas 0.4.3 imports a langchain-community module that newer releases removed, so the second command pins that package. Keep --no-deps so pip doesn’t downgrade langchain-core. pip check then reports that langchain-community requires dataclasses-json. Ragas still runs, so you can ignore that warning.

Export your SambaCloud API key

export SAMBANOVA_API_KEY="your-sambanova-api-key"

Point Ragas's judge model at SambaCloud

Ragas’s llm_factory accepts any OpenAI client instance, including one configured with a custom base_url:

import os
from openai import OpenAI
from ragas.llms import llm_factory

client = OpenAI(
    api_key=os.environ["SAMBANOVA_API_KEY"],
    base_url="https://api.sambanova.ai/v1",
)

llm = llm_factory("MiniMax-M3.1-Flash", client=client)

Metrics

Ragas groups its LLM-judged metrics into three output shapes. Each example below uses the llm from Setup.

Discrete: a correct/incorrect verdict

Score a correct and an incorrect answer to check that the judge tells them apart:

from ragas.metrics import DiscreteMetric

metric = DiscreteMetric(
    name="correctness",
    prompt="Given the question and response, judge if the response correctly answers the question. Question: {query} Response: {response}. Return 'correct' or 'incorrect'.",
    allowed_values=["correct", "incorrect"],
)

# A correct answer
result = metric.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Paris.",
)
print(result.value, "-", result.reason)

# An incorrect answer
bad_result = metric.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Berlin.",
)
print(bad_result.value, "-", bad_result.reason)

Output

correct - The response correctly identifies Paris as the capital of France.
incorrect - The capital of France is Paris, not Berlin.

Numeric: a score on a continuous scale

from ragas.metrics import NumericMetric

relevance = NumericMetric(
    name="relevance",
    allowed_values=(0, 10),
    prompt="Rate how relevant the response is to the query on a scale of 0 to 10. Query: {query} Response: {response}.",
)
result = relevance.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Paris, a city known for the Eiffel Tower.",
)
print(result.value, "-", result.reason)

Output

10.0 - The response directly and correctly identifies Paris as the capital of France. The additional detail about the Eiffel Tower does not detract from its relevance.

Ranking: ordering multiple candidate responses

from ragas.metrics import RankingMetric

quality_ranking = RankingMetric(
    name="quality_ranking",
    allowed_values=3,
    prompt="Given the query and a list of responses, return the indices of the responses ranked from best to worst. Query: {query} Responses: {responses}.",
)
result = quality_ranking.score(
    llm=llm,
    query="What is the capital of France?",
    responses=[
        "I don't know.",
        "France's capital is a city in Europe.",
        "The capital of France is Paris.",
    ],
)
print(result.value, "-", result.reason)

Output

['2', '1', '0'] - Response 2 directly and correctly states that the capital of France is Paris. Response 1 is vague but provides some relevant information, while Response 0 does not answer the question.

Datasets and experiments

Ragas’s Dataset and @experiment give you a repeatable way to run a batch of inputs through your system and score every result, instead of calling a metric one example at a time. This example uses client and llm from Setup and metric from the Discrete example.

import asyncio
from ragas import Dataset, experiment

dataset = Dataset(name="capitals", backend="local/csv", root_dir="./ragas_data")
dataset.append({"id": "1", "query": "What is the capital of France?", "expected_answer": "Paris"})
dataset.append({"id": "2", "query": "What is the capital of Japan?", "expected_answer": "Tokyo"})
dataset.save()


@experiment()
async def answer_and_score(row):
    completion = client.chat.completions.create(
        model="MiniMax-M3.1-Flash",
        messages=[{"role": "user", "content": row["query"]}],
    )
    response = completion.choices[0].message.content
    score = metric.score(llm=llm, query=row["query"], response=response)
    return {**row, "response": response, "correctness": score.value, "reason": score.reason}


async def main():
    results = await answer_and_score.arun(dataset)
    for r in sorted(results, key=lambda r: r["id"]):
        print(r["id"], "->", r["correctness"], "-", r["response"][:60])


asyncio.run(main())

Output

1 -> correct - The capital of France is **Paris**.
2 -> correct - The capital of Japan is **Tokyo**.

Each row runs one SambaCloud chat completion and one DiscreteMetric judge call. dataset.save() writes ragas_data/datasets/capitals.csv, and arun() writes the scored results to a CSV with a randomly generated name under ragas_data/experiments/, so you can compare experiment runs later.

Not covered: synthetic test data generation

Ragas’s RAG test set generation (TestsetGenerator) builds synthetic queries and reference answers from your own documents via a knowledge graph. Unlike every example above, it requires both an LLM and an embedding model:

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)

SambaCloud has no embedding model, so this feature can’t run on SambaCloud alone. Embedding models such as E5-Mistral-7B-Instruct are available on SambaStack. See Embeddings. If you need synthetic test data generation, pair your SambaStack instance’s embedding model with a SambaCloud or SambaStack LLM for this one feature, and use SambaCloud for everything else on this page.

Troubleshooting

ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'

You skipped the pinned langchain-community install in Setup. Run pip install "langchain-community==0.3.30" --no-deps.

KeyError: SAMBANOVA_API_KEY

Export the key before running: export SAMBANOVA_API_KEY="your-key". Verify with echo $SAMBANOVA_API_KEY.

Judge always returns the same value regardless of input

Test both a correct and a deliberately incorrect response against your metric before trusting it in a real eval run, as shown in the Discrete example above. A judge that doesn’t discriminate is not providing a useful signal, whatever its stated score.

TestsetGenerator raises an error with no embedding model configured

Expected: synthetic test data generation needs an embedding model, which SambaCloud doesn’t provide. See Not covered above and use your SambaStack instance’s embedding model for this feature specifically.

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

Ragas integration guide

Prerequisites

Setup

Metrics

Discrete: a correct/incorrect verdict

Numeric: a score on a continuous scale

Ranking: ordering multiple candidate responses

Datasets and experiments

Not covered: synthetic test data generation

Troubleshooting

Additional resources

​Prerequisites

​Setup

​Metrics

​Discrete: a correct/incorrect verdict

​Numeric: a score on a continuous scale

​Ranking: ordering multiple candidate responses

​Datasets and experiments

​Not covered: synthetic test data generation

​Troubleshooting

​Additional resources

Prerequisites

Setup

Metrics

Discrete: a correct/incorrect verdict

Numeric: a score on a continuous scale

Ranking: ordering multiple candidate responses

Datasets and experiments

Not covered: synthetic test data generation

Troubleshooting

Additional resources