# Evaluation

Run standardized benchmarks against any Tinker model or checkpoint. The framework handles concurrency, grading, trajectory storage, and result aggregation.

## Quick Start

```python
from tinker_cookbook.eval.benchmarks import run_benchmark, run_benchmarks, BenchmarkConfig

# Single benchmark
result = await run_benchmark("gsm8k", sampling_client, renderer)
print(f"GSM8K: {result.score:.1%}")

# Multiple benchmarks (parallel by default)
results = await run_benchmarks(
    ["gsm8k", "mmlu_pro", "ifeval"],
    sampling_client, renderer,
)
for name, r in results.items():
    print(f"{name}: {r.score:.1%}")
```

## Architecture

Benchmarks reuse the RL `Env` protocol — the same environment can drive both evaluation and training.

```text
BenchmarkBuilder
├── make_envs(renderer, config) → list[Env]   one Env per example
├── aggregate(rewards, metrics) → BenchmarkResult
└── name, multi_turn, requires_sandbox, ...

Runner (run_benchmark / run_benchmarks)
├── Creates envs from BenchmarkBuilder
├── Runs rollouts concurrently (semaphore)
├── Grades responses via Env.step()
├── Saves trajectories as JSONL
└── Aggregates into BenchmarkResult

BenchmarkResult
├── score              raw accuracy
├── score_completed    accuracy excluding truncated/errored examples
├── num_examples, num_correct, num_errors, num_truncated
├── pass_at_k          {1: 0.45, 5: 0.72}  (when num_samples > 1)
└── time_seconds
```

## Available Benchmarks

### Programmatic Grading (Single-Turn)

| Benchmark       | Examples | Description                                                                                                    |
| --------------- | -------- | -------------------------------------------------------------------------------------------------------------- |
| `gsm8k`         | 1,319    | [Grade school math](https://huggingface.co/datasets/openai/gsm8k)                                              |
| `math500`       | 500      | [Competition math (MATH)](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)                              |
| `aime`          | 30       | [AIME math competition](https://huggingface.co/datasets/MathArena/aime_2025)                                   |
| `aime_2025`     | 30       | [AIME 2025](https://huggingface.co/datasets/MathArena/aime_2025)                                               |
| `aime_2026`     | 30       | [AIME 2026](https://huggingface.co/datasets/MathArena/aime_2026)                                               |
| `hmmt_feb_2025` | 30       | [HMMT February 2025](https://huggingface.co/datasets/MathArena/hmmt_feb_2025)                                  |
| `hmmt_nov_2025` | 30       | [HMMT November 2025](https://huggingface.co/datasets/MathArena/hmmt_nov_2025)                                  |
| `mmlu_pro`      | 12,032   | [Multi-domain knowledge](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro)                                   |
| `mmlu_redux`    | ~3,000   | [Curated MMLU subset](https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux) (30 subjects, error-filtered) |
| `gpqa`          | 198      | [Graduate-level science QA](https://huggingface.co/datasets/Idavidrein/gpqa) (Diamond set)                     |
| `supergpqa`     | 26,529   | [Extended graduate QA](https://arxiv.org/abs/2502.14739)                                                       |
| `ceval`         | ~1,300   | [Chinese exam evaluation](https://huggingface.co/datasets/ceval/ceval-exam) (52 subjects, val split)           |
| `ifeval`        | 541      | [Instruction following](https://huggingface.co/datasets/google/IFEval)                                         |
| `ifbench`       | 300      | [Instruction following](https://huggingface.co/datasets/allenai/IFBench_test)                                  |

### Code Execution

| Benchmark        | Type        | Description                                                                                              |
| ---------------- | ----------- | -------------------------------------------------------------------------------------------------------- |
| `mbpp`           | Single-turn | [Python code generation](https://huggingface.co/datasets/google-research-datasets/mbpp) (sanitized, 427) |
| `livecodebench`  | Single-turn | [Competitive programming](https://huggingface.co/datasets/livecodebench/code_generation_lite)            |
| `swe_bench`      | Multi-turn  | [GitHub issue resolution](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified) (500, agent) |
| `terminal_bench` | Multi-turn  | [Terminal interaction](https://huggingface.co/datasets/ia03/terminal-bench) (agent)                      |
| `tau2_bench`     | Multi-turn  | [Customer service agent](https://github.com/sierra-research/tau2-bench)                                  |

### LLM-as-Judge

| Benchmark    | Description                                                                                                          |
| ------------ | -------------------------------------------------------------------------------------------------------------------- |
| `arena_hard` | [ArenaHard](https://huggingface.co/datasets/lmarena-ai/arena-hard-auto-v0.1) (500, requires `judge_sampling_client`) |

### Other

| Benchmark   | Description                                                                                                      |
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| `longbench` | [Long-context evaluation](https://huggingface.co/datasets/THUDM/LongBench-v2)                                    |
| `bfcl`      | [Function calling / tool use](https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard) |

## Using Benchmarks During Training

`BenchmarkEvaluator` wraps any benchmark as a training-loop evaluator. It runs a small sample at each eval step and logs metrics to TensorBoard / W&B:

```python
from tinker_cookbook.eval.benchmark_evaluator import BenchmarkEvaluator

config = train.Config(
    # ... training config ...
    evaluator_builders=[
        lambda: BenchmarkEvaluator("gsm8k", renderer, max_examples=100),
        lambda: BenchmarkEvaluator("ifeval", renderer, max_examples=50),
    ],
    eval_every=50,
)
```

Logged metrics: `eval/gsm8k/score`, `eval/gsm8k/num_correct`, etc.

## Next Steps

- **[Benchmarks Guide](https://tinker-docs.thinkingmachines.ai/cookbook/eval/benchmarks-guide/index.md)** — configuration, storage, pass@k, adding new benchmarks
- **[Customizing Benchmarks](https://tinker-docs.thinkingmachines.ai/cookbook/eval/customization/index.md)** — system prompts, answer parsing, judge models, shared Env design
- **[Evaluations Tutorial](https://tinker-docs.thinkingmachines.ai/tutorials/core-concepts/evaluations/index.md)** — interactive walkthrough of the evaluator pattern
- **[API Reference](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/eval/index.md)** — full type documentation