> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LM Evaluation Harness integration guide

[Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) is a unified evaluation framework developed by [EleutherAI](https://www.eleuther.ai/) for testing generative language models across a wide range of evaluation tasks.

## Features

* Supports over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.
* Compatible with commercial APIs.
* Supports local models and custom benchmarks.
* Uses publicly available prompts to ensure reproducibility and comparability between papers.
* Easy integration of custom prompts and evaluation metrics.

## Prerequisites

* Python 3.9 or later
* [Git](https://git-scm.com/)

## Setup

<Steps>
  <Step title="Create a SambaCloud account">
    Create a [SambaCloud](https://cloud.sambanova.ai/?utm_source=lmevaluationharness\&utm_medium=external\&utm_campaign=cloud_signup) account and obtain an [API key](https://cloud.sambanova.ai/apis?utm_source=lmevaluationharness\&utm_medium=external\&utm_campaign=cloud_signup).
  </Step>

  <Step title="Clone the repository">
    ```bash theme={}
    git clone https://github.com/EleutherAI/lm-evaluation-harness.git
    cd lm-evaluation-harness
    ```
  </Step>

  <Step title="Create and activate a virtual environment">
    ```bash theme={}
    python -m venv .venv
    source .venv/bin/activate
    ```
  </Step>

  <Step title="Install dependencies">
    ```bash theme={}
    pip install -e .
    pip install -e ."[api]"
    pip install tqdm
    ```

    <Note>
      Additional Python packages may be required depending on the selected benchmark or task. If you encounter errors related to missing libraries, install them manually.
    </Note>
  </Step>

  <Step title="Set your API key">
    ```bash theme={}
    export SAMBANOVA_API_KEY="your-sambanova-api-key"
    ```
  </Step>
</Steps>

## Example use case

* Example benchmark: GSM8K (Grade School Math)
* Model source: SambaCloud

Run the evaluation against a SambaCloud model:

```bash theme={}
lm_eval \
  --model openai-chat-completions \
  --model_args model=Meta-Llama-3.3-70B-Instruct,base_url=https://api.sambanova.ai/v1,token="$SAMBANOVA_API_KEY" \
  --tasks gsm8k \
  --num_fewshot 5 \
  --limit 100
```

This example evaluates reasoning and arithmetic skills using standard prompt formats and metrics. Replace `Meta-Llama-3.3-70B-Instruct` with any model ID from the [SambaCloud models](/docs/en/models/sambacloud-models) page.

### Resources

* [Quickstart Notebook](https://github.com/sambanova/integrations/blob/main/lm_eval_harness/quickstart_eval.ipynb)
* [Integration directory](https://github.com/sambanova/integrations/tree/main/lm_eval_harness)

## Troubleshooting

<AccordionGroup>
  <Accordion title="Authentication error or SAMBANOVA_API_KEY not recognized">
    Confirm the variable is exported in your current shell:

    ```bash theme={}
    echo $SAMBANOVA_API_KEY
    ```

    Re-export if empty: `export SAMBANOVA_API_KEY="your-api-key"`. Verify the key is active in the [SambaCloud portal](https://cloud.sambanova.ai/apis?utm_source=lmevaluationharness\&utm_medium=external\&utm_campaign=cloud_signup).
  </Accordion>

  <Accordion title="Task not found">
    Check available task names with:

    ```bash theme={}
    lm_eval --tasks list
    ```

    Confirm you installed the `[api]` extras: `pip install -e ."[api]"`.
  </Accordion>

  <Accordion title="Missing library errors during evaluation">
    Some benchmarks require additional packages. Install them as reported in the error, for example:

    ```bash theme={}
    pip install sqlitedict pytablewriter
    ```
  </Accordion>
</AccordionGroup>