> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ragas integration guide

[Ragas](https://docs.ragas.io/) is a metric library for evaluating RAG pipelines. Judge-model calls are the core of how it scores responses, so a fast judge directly speeds up every evaluation run.

## Prerequisites

* A [SambaCloud](https://cloud.sambanova.ai/?utm_source=ragas\&utm_medium=external\&utm_campaign=cloud_signup) account and [API key](https://cloud.sambanova.ai/apis?utm_source=ragas\&utm_medium=external\&utm_campaign=cloud_signup)
* Python 3.9 or later

## Setup

**1. Install Ragas**

```bash theme={}
pip install "ragas==0.4.3" openai
pip install "langchain-community==0.3.30" --no-deps
```

> [!NOTE]
>
> Ragas 0.4.3 imports a `langchain-community` module that newer releases removed, so the second command pins that package. Keep `--no-deps` so pip doesn't downgrade `langchain-core`. `pip check` then reports that `langchain-community` requires `dataclasses-json`. Ragas still runs, so you can ignore that warning.

**2. Export your SambaCloud API key**

```bash theme={}
export SAMBANOVA_API_KEY="your-sambanova-api-key"
```

**3. Point Ragas's judge model at SambaCloud**

Ragas's `llm_factory` accepts any OpenAI client instance, including one configured with a custom `base_url`:

```python theme={}
import os
from openai import OpenAI
from ragas.llms import llm_factory

client = OpenAI(
    api_key=os.environ["SAMBANOVA_API_KEY"],
    base_url="https://api.sambanova.ai/v1",
)

llm = llm_factory("MiniMax-M3.1-Flash", client=client)
```

## Metrics

Ragas groups its LLM-judged metrics into three output shapes. Each example below uses the `llm` from Setup.

### Discrete: a correct/incorrect verdict

Score a correct and an incorrect answer to check that the judge tells them apart:

```python theme={}
from ragas.metrics import DiscreteMetric

metric = DiscreteMetric(
    name="correctness",
    prompt="Given the question and response, judge if the response correctly answers the question. Question: {query} Response: {response}. Return 'correct' or 'incorrect'.",
    allowed_values=["correct", "incorrect"],
)

# A correct answer
result = metric.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Paris.",
)
print(result.value, "-", result.reason)

# An incorrect answer
bad_result = metric.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Berlin.",
)
print(bad_result.value, "-", bad_result.reason)
```

```text Output theme={}
correct - The response correctly identifies Paris as the capital of France.
incorrect - The capital of France is Paris, not Berlin.
```

### Numeric: a score on a continuous scale

```python theme={}
from ragas.metrics import NumericMetric

relevance = NumericMetric(
    name="relevance",
    allowed_values=(0, 10),
    prompt="Rate how relevant the response is to the query on a scale of 0 to 10. Query: {query} Response: {response}.",
)
result = relevance.score(
    llm=llm,
    query="What is the capital of France?",
    response="The capital of France is Paris, a city known for the Eiffel Tower.",
)
print(result.value, "-", result.reason)
```

```text Output theme={}
10.0 - The response directly and correctly identifies Paris as the capital of France. The additional detail about the Eiffel Tower does not detract from its relevance.
```

### Ranking: ordering multiple candidate responses

```python theme={}
from ragas.metrics import RankingMetric

quality_ranking = RankingMetric(
    name="quality_ranking",
    allowed_values=3,
    prompt="Given the query and a list of responses, return the indices of the responses ranked from best to worst. Query: {query} Responses: {responses}.",
)
result = quality_ranking.score(
    llm=llm,
    query="What is the capital of France?",
    responses=[
        "I don't know.",
        "France's capital is a city in Europe.",
        "The capital of France is Paris.",
    ],
)
print(result.value, "-", result.reason)
```

```text Output theme={}
['2', '1', '0'] - Response 2 directly and correctly states that the capital of France is Paris. Response 1 is vague but provides some relevant information, while Response 0 does not answer the question.
```

## Datasets and experiments

Ragas's `Dataset` and `@experiment` give you a repeatable way to run a batch of inputs through your system and score every result, instead of calling a metric one example at a time. This example uses `client` and `llm` from Setup and `metric` from the Discrete example.

```python theme={}
import asyncio
from ragas import Dataset, experiment

dataset = Dataset(name="capitals", backend="local/csv", root_dir="./ragas_data")
dataset.append({"id": "1", "query": "What is the capital of France?", "expected_answer": "Paris"})
dataset.append({"id": "2", "query": "What is the capital of Japan?", "expected_answer": "Tokyo"})
dataset.save()


@experiment()
async def answer_and_score(row):
    completion = client.chat.completions.create(
        model="MiniMax-M3.1-Flash",
        messages=[{"role": "user", "content": row["query"]}],
    )
    response = completion.choices[0].message.content
    score = metric.score(llm=llm, query=row["query"], response=response)
    return {**row, "response": response, "correctness": score.value, "reason": score.reason}


async def main():
    results = await answer_and_score.arun(dataset)
    for r in sorted(results, key=lambda r: r["id"]):
        print(r["id"], "->", r["correctness"], "-", r["response"][:60])


asyncio.run(main())
```

```text Output theme={}
1 -> correct - The capital of France is **Paris**.
2 -> correct - The capital of Japan is **Tokyo**.
```

Each row runs one SambaCloud chat completion and one `DiscreteMetric` judge call. `dataset.save()` writes `ragas_data/datasets/capitals.csv`, and `arun()` writes the scored results to a CSV with a randomly generated name under `ragas_data/experiments/`, so you can compare experiment runs later.

## Not covered: synthetic test data generation

Ragas's RAG test set generation (`TestsetGenerator`) builds synthetic queries and reference answers from your own documents via a knowledge graph. Unlike every example above, it requires **both** an LLM and an embedding model:

```python theme={}
from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
```

SambaCloud has no embedding model, so this feature can't run on SambaCloud alone. Embedding models such as `E5-Mistral-7B-Instruct` are available on SambaStack. See [Embeddings](https://docs.sambanova.ai/docs/en/features/embeddings.md). If you need synthetic test data generation, pair your SambaStack instance's embedding model with a SambaCloud or SambaStack LLM for this one feature, and use SambaCloud for everything else on this page.

## Troubleshooting

> [accordion] ModuleNotFoundError: No module named 'langchain\_community.chat\_models.vertexai'
>
> You skipped the pinned `langchain-community` install in [Setup](#setup). Run `pip install "langchain-community==0.3.30" --no-deps`.

> [accordion] KeyError: SAMBANOVA\_API\_KEY
>
> Export the key before running: `export SAMBANOVA_API_KEY="your-key"`. Verify with `echo $SAMBANOVA_API_KEY`.

> [accordion] Judge always returns the same value regardless of input
>
> Test both a correct and a deliberately incorrect response against your metric before trusting it in a real eval run, as shown in the Discrete example above. A judge that doesn't discriminate is not providing a useful signal, whatever its stated score.

> [accordion] TestsetGenerator raises an error with no embedding model configured
>
> Expected: synthetic test data generation needs an embedding model, which SambaCloud doesn't provide. See [Not covered](#not-covered-synthetic-test-data-generation) above and use your SambaStack instance's embedding model for this feature specifically.

## Additional resources

* [Ragas documentation](https://docs.ragas.io/)
* [Ragas metrics overview](https://docs.ragas.io/en/stable/concepts/metrics/overview/)
* [Ragas datasets](https://docs.ragas.io/en/stable/concepts/datasets/)
* [Ragas experimentation](https://docs.ragas.io/en/stable/concepts/experimentation/)
* [Customizing models in Ragas](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/)
* [Supported SambaCloud models](https://docs.sambanova.ai/docs/en/models/sambacloud-models.md)