Build with LM Evaluation Harness - SambaNova Integration Guide

Language Model Evaluation Harness is a unified evaluation framework developed by EleutherAI for testing generative language models across a wide range of evaluation tasks.

Features

Supports over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.
Compatible with commercial APIs.
Supports local models and custom benchmarks.
Uses publicly available prompts to ensure reproducibility and comparability between papers.
Easy integration of custom prompts and evaluation metrics.

Prerequisites

Python 3.9 or later
Git

Setup

Create a SambaCloud account

Create a SambaCloud account and obtain an API key.

Clone the repository

git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness

Create and activate a virtual environment

python -m venv .venv
source .venv/bin/activate

Install dependencies

pip install -e .
pip install -e ."[api]"
pip install tqdm

Additional Python packages may be required depending on the selected benchmark or task. If you encounter errors related to missing libraries, install them manually.

Set your API key

export SAMBANOVA_API_KEY="your-sambanova-api-key"

Example use case

Example benchmark: GSM8K (Grade School Math)
Model source: SambaCloud

Run the evaluation against a SambaCloud model:

lm_eval \
  --model openai-chat-completions \
  --model_args model=Meta-Llama-3.3-70B-Instruct,base_url=https://api.sambanova.ai/v1,token="$SAMBANOVA_API_KEY" \
  --tasks gsm8k \
  --num_fewshot 5 \
  --limit 100

This example evaluates reasoning and arithmetic skills using standard prompt formats and metrics. Replace Meta-Llama-3.3-70B-Instruct with any model ID from the SambaCloud models page.

Resources

Troubleshooting

Authentication error or SAMBANOVA_API_KEY not recognized

Confirm the variable is exported in your current shell:

echo $SAMBANOVA_API_KEY

Re-export if empty: export SAMBANOVA_API_KEY="your-api-key". Verify the key is active in the SambaCloud portal.

Task not found

Check available task names with:

lm_eval --tasks list

Confirm you installed the [api] extras: pip install -e ."[api]".

Missing library errors during evaluation

Some benchmarks require additional packages. Install them as reported in the error, for example:

pip install sqlitedict pytablewriter

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

LM Evaluation Harness integration guide

Features

Prerequisites

Setup

Example use case

Resources

Troubleshooting

​Features

​Prerequisites

​Setup

​Example use case

​Resources

​Troubleshooting

Features

Prerequisites

Setup

Example use case

Resources

Troubleshooting