Build with DeepEval - SambaNova Integration Guide

DeepEval is an open-source LLM evaluation framework that plugs into pytest, so judge-model calls run as part of your normal test suite and CI pipeline.

Prerequisites

A SambaCloud account and API key
Python 3.9 or later

Setup

Install DeepEval

pip install deepeval pytest

Export your SambaCloud API key

export SAMBANOVA_API_KEY="your-sambanova-api-key"

Point DeepEval's judge model at SambaCloud

Use DeepEval’s OpenAIModel class with a custom base_url — this is the same class DeepEval uses for real OpenAI, just redirected:

import os
from deepeval.models import OpenAIModel

model = OpenAIModel(
    model="MiniMax-M3",
    api_key=os.environ["SAMBANOVA_API_KEY"],
    base_url="https://api.sambanova.ai/v1",
)

See DeepEval’s OpenAI model reference for the full parameter list. The older GPTModel name works identically but is deprecated in favor of OpenAIModel as of DeepEval 4.x.

Example: score a test case

test_sambanova.py

import os
from deepeval.models import OpenAIModel
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

model = OpenAIModel(
    model="MiniMax-M3",
    api_key=os.environ["SAMBANOVA_API_KEY"],
    base_url="https://api.sambanova.ai/v1",
)

test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris.",
)

metric = AnswerRelevancyMetric(model=model, threshold=0.5)
metric.measure(test_case)
print(metric.score)
print(metric.reason)

Output

1.0
The score is 1.00 because the response directly and accurately addresses the question about the capital of France with no irrelevant information. Excellent work!

The reason string is generated by the judge model, so it explains the score rather than restating it.

Example: run as a pytest CI check

This is DeepEval’s main use case: judge-model calls inside your normal test suite.

test_ci_example.py

import os
from deepeval import assert_test
from deepeval.models import OpenAIModel
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

model = OpenAIModel(
    model="MiniMax-M3",
    api_key=os.environ["SAMBANOVA_API_KEY"],
    base_url="https://api.sambanova.ai/v1",
)

def test_answer_relevancy():
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
    )
    metric = AnswerRelevancyMetric(model=model, threshold=0.5)
    assert_test(test_case, [metric])

Run it with DeepEval’s own CLI (its documented command, backed by pytest under the hood):

deepeval test run test_ci_example.py

Output (abridged)

✓ Evaluation completed 🎉! (time taken: 5.44s | token cost: None)
» Test Results (1 total tests):
   » Pass Rate: 100.0% | Passed: 1 | Failed: 0

Plain pytest test_ci_example.py -v also works, since DeepEval is a pytest plugin, but deepeval test run is the documented command and adds the richer test-result table shown above.

DeepEval itself needs no login or API key to run evaluations locally — deepeval login is optional and only needed for Confident AI’s cloud dashboard. The only credential required for the examples on this page is your SambaCloud API key, passed to OpenAIModel.

Troubleshooting

DeprecationWarning: 'GPTModel' is deprecated

Use OpenAIModel instead — same constructor, same base_url support, just the current name as of DeepEval 4.x.

KeyError: SAMBANOVA_API_KEY

Export the key before running: export SAMBANOVA_API_KEY="your-key". Verify with echo $SAMBANOVA_API_KEY.

Low or unexpected AnswerRelevancyMetric scores

Check metric.reason — it’s generated by the judge model itself and explains exactly what it penalized, which is more useful than the score alone for figuring out whether the issue is your prompt, your test case, or the judge model’s interpretation.

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

DeepEval integration guide

Prerequisites

Setup

Example: score a test case

Example: run as a pytest CI check

Troubleshooting

Additional resources

​Prerequisites

​Setup

​Example: score a test case

​Example: run as a pytest CI check

​Troubleshooting

​Additional resources

Prerequisites

Setup

Example: score a test case

Example: run as a pytest CI check

Troubleshooting

Additional resources