# Tutorial 205: Evaluations

> **Prerequisites**
>
> - [Your First SFT](https://tinker-docs.thinkingmachines.ai/tutorials/basics/first-sft/index.md)

> **Run it interactively [[source]](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tutorials/205_evaluations.py)**
>
> ```bash
> curl -O https://raw.githubusercontent.com/thinking-machines-lab/tinker-cookbook/main/tutorials/205_evaluations.py && marimo edit 205_evaluations.py
> ```

Tinker's evaluation system uses two abstract classes:

- **`TrainingClientEvaluator`** -- uses the training client (forward passes) to compute metrics like NLL
- **`SamplingClientEvaluator`** -- uses the sampling client (generation) to compute metrics like accuracy

Both return `dict[str, float]` and plug into `train.Config.evaluator_builders` for automatic evaluation during training.

In this tutorial you will:

1. Implement a `TrainingClientEvaluator` that computes NLL on held-out data
1. Implement a `SamplingClientEvaluator` that samples answers and checks correctness
1. Wire evaluators into `train.Config` via `evaluator_builders`
1. Learn about the Inspect AI integration for standardized benchmarks

```python
import warnings

warnings.filterwarnings("ignore", message="IProgress not found")

import tinker
import torch
from tinker import TensorData

from tinker_cookbook.eval.evaluators import (
    SamplingClientEvaluator,
    TrainingClientEvaluator,
)
from tinker_cookbook.renderers import get_renderer, get_text_content
```

## The evaluator pattern

Both evaluator types are async callables with a simple contract:

```python
class TrainingClientEvaluator:
    async def __call__(self, training_client: tinker.TrainingClient) -> dict[str, float]:
        ...

class SamplingClientEvaluator:
    async def __call__(self, sampling_client: tinker.SamplingClient) -> dict[str, float]:
        ...
```

The training loop calls your evaluator periodically and logs the returned metrics. The keys become metric names (e.g., `"eval/nll"`, `"eval/accuracy"`).

## Setup

Create a training client and prepare some evaluation data.

```python
api_key = mo.ui.text(kind="password", label="Paste your Tinker API key")
api_key  # noqa: B018
```

```python
import os

mo.stop(
    "TINKER_API_KEY" not in os.environ and not api_key.value,
    "Paste your API key above",
)

if api_key.value:
    os.environ["TINKER_API_KEY"] = api_key.value

MODEL_NAME = "Qwen/Qwen3.5-4B"

service_client = tinker.ServiceClient()
training_client = await service_client.create_lora_training_client_async(
    base_model=MODEL_NAME, rank=16
)
tokenizer = training_client.get_tokenizer()
renderer = get_renderer("qwen3_5_disable_thinking", tokenizer)

# Prepare held-out SFT data for the NLL evaluator
eval_examples = [
    "The speed of light is approximately 3 * 10^8 meters per second.",
    "Water freezes at 0 degrees Celsius under standard pressure.",
    "The Earth orbits the Sun once every 365.25 days.",
]

eval_datums = []
for text in eval_examples:
    ids = tokenizer.encode(text)
    model_input = tinker.ModelInput.from_ints(ids[:-1])
    target_tokens = ids[1:]
    w = [1.0] * len(target_tokens)
    eval_datums.append(
        tinker.Datum(
            model_input=model_input,
            loss_fn_inputs={
                "target_tokens": TensorData.from_torch(torch.tensor(target_tokens)),
                "weights": TensorData.from_torch(torch.tensor(w)),
            },
        )
    )

print(f"Prepared {len(eval_datums)} evaluation datums")
```

**Output**

```text
Prepared 3 evaluation datums
```

## Implementing a TrainingClientEvaluator: NLL

A `TrainingClientEvaluator` receives the current `TrainingClient` and can run forward passes to compute metrics. Here we compute the mean negative log-likelihood (NLL) on held-out data -- a standard measure of how well the model predicts the evaluation text.

```python
class NLLEvaluator(TrainingClientEvaluator):
    """Compute mean NLL on held-out data using forward passes."""

    def __init__(self, eval_data: list[tinker.Datum], name: str = "eval"):
        self.eval_data = eval_data
        self.name = name

    async def __call__(self, training_client: tinker.TrainingClient) -> dict[str, float]:
        # Run a forward pass (no gradients) to get logprobs
        future = await training_client.forward_async(self.eval_data, loss_fn="cross_entropy")
        result = await future.result_async()

        # Compute weighted mean NLL
        total_nll = 0.0
        total_tokens = 0
        for datum, output in zip(self.eval_data, result.loss_fn_outputs):
            logprobs = output["logprobs"].to_torch()
            weights = datum.loss_fn_inputs["weights"].to_torch()
            total_nll += -(logprobs * weights).sum().item()
            total_tokens += weights.sum().item()

        mean_nll = total_nll / max(total_tokens, 1)
        return {f"{self.name}/nll": mean_nll}
```

```python
# Test the evaluator
nll_evaluator = NLLEvaluator(eval_datums, name="held_out")
nll_metrics = await nll_evaluator(training_client)
print(f"NLL evaluation: {nll_metrics}")
```

**Output**

```text
NLL evaluation: {'held_out/nll': 1.3247994922456288}
```

## Implementing a SamplingClientEvaluator: Accuracy

A `SamplingClientEvaluator` receives a `SamplingClient` and generates text to compute metrics. Here we sample answers to simple factual questions and check if they contain the expected answer.

```python
class AccuracyEvaluator(SamplingClientEvaluator):
    """Sample answers and check if they contain the expected string."""

    def __init__(self, questions_and_answers, renderer, tokenizer):
        self.qa_pairs = questions_and_answers
        self.renderer = renderer
        self.tokenizer = tokenizer

    async def __call__(self, sampling_client: tinker.SamplingClient) -> dict[str, float]:
        correct = 0

        for question, expected_answer in self.qa_pairs:
            messages = [{"role": "user", "content": question}]
            prompt = self.renderer.build_generation_prompt(messages)
            stop = self.renderer.get_stop_sequences()

            result = await sampling_client.sample_async(
                prompt=prompt,
                sampling_params=tinker.SamplingParams(
                    max_tokens=64, temperature=0.0, stop=stop
                ),
                num_samples=1,
            )

            tokens = result.sequences[0].tokens
            parsed, _ = self.renderer.parse_response(tokens)
            response_text = get_text_content(parsed).lower()

            if expected_answer.lower() in response_text:
                correct += 1

        accuracy = correct / len(self.qa_pairs)
        return {"eval/accuracy": accuracy, "eval/correct": float(correct)}
```

```python
# Create a sampling client from current weights
sampling_client = await training_client.save_weights_and_get_sampling_client_async()

# Define test questions
test_qa = [
    ("What is the capital of Japan?", "tokyo"),
    ("What is 15 + 27?", "42"),
    ("What element has the symbol 'O'?", "oxygen"),
]

accuracy_evaluator = AccuracyEvaluator(test_qa, renderer, tokenizer)
acc_metrics = await accuracy_evaluator(sampling_client)
print(f"Accuracy evaluation: {acc_metrics}")
```

**Output**

```text
Accuracy evaluation: {'eval/accuracy': 0.6666666666666666, 'eval/correct': 2.0}
```

## Wiring evaluators into train.Config

The `train.Config` classes in `tinker_cookbook.supervised.train` and `tinker_cookbook.rl.train` accept an `evaluator_builders` parameter. Each builder is a zero-argument callable that returns an evaluator instance.

The training loop calls each builder once at startup, then runs the evaluators periodically (controlled by `eval_every`).

### Supervised training with evaluators

```python
from tinker_cookbook.supervised import train

def make_nll_evaluator():
    # Build eval data here (or capture it from outer scope)
    return NLLEvaluator(eval_datums, name="validation")

def make_accuracy_evaluator():
    return AccuracyEvaluator(test_qa, renderer, tokenizer)

config = train.Config(
    log_path="/tmp/tinker-tutorials/sft-with-evals",
    model_name="Qwen/Qwen3.5-4B",
    dataset_builder=my_dataset_builder,
    learning_rate=1e-4,

    # Evaluator builders -- called once at startup, run every eval_every steps
    evaluator_builders=[make_nll_evaluator, make_accuracy_evaluator],
    eval_every=50,  # run evaluators every 50 steps
)
```

The SFT training loop detects the evaluator type automatically:

- `TrainingClientEvaluator` is called with the training client
- `SamplingClientEvaluator` is called with a sampling client created from current weights

### RL training with evaluators

The RL `train.Config` works the same way, though it only accepts `SamplingClientEvaluatorBuilder`:

```python
from tinker_cookbook.rl import train

config = train.Config(
    log_path="/tmp/tinker-tutorials/rl-with-evals",
    model_name="Qwen/Qwen3.5-9B-Base",
    dataset_builder=my_rl_dataset_builder,

    evaluator_builders=[make_accuracy_evaluator],
    eval_every=10,
)
```

## Built-in evaluator: NLLEvaluator

The cookbook ships a production-ready NLL evaluator at `tinker_cookbook.supervised.nll_evaluator.NLLEvaluator`. It can be constructed from a `SupervisedDataset`:

```python
from tinker_cookbook.supervised.nll_evaluator import NLLEvaluator

# From a dataset object
nll_eval = NLLEvaluator.from_dataset(eval_dataset, name="test")

# Or from raw datums
nll_eval = NLLEvaluator(data=eval_datums, name="validation")
```

## Inspect AI integration

For standardized benchmarks (MMLU, GSM8K, HumanEval, etc.), the cookbook integrates with [Inspect AI](https://inspect.aisi.org.uk/). The `InspectEvaluator` wraps any Inspect task as a `SamplingClientEvaluator`:

```python
from tinker_cookbook.eval.run_inspect_evals import Config

# Run Inspect evals standalone
config = Config(
    model_path="tinker://run-id/sampler_weights/final",
    # ... Inspect task configuration ...
)
```

Inspect AI provides a large library of pre-built evaluation tasks, so you can benchmark your fine-tuned model against established benchmarks without writing custom evaluation code.

## Summary

| Evaluator type            | Receives         | Typical metrics  | Example                       |
| ------------------------- | ---------------- | ---------------- | ----------------------------- |
| `TrainingClientEvaluator` | `TrainingClient` | NLL, perplexity  | Forward pass on held-out data |
| `SamplingClientEvaluator` | `SamplingClient` | Accuracy, reward | Generate and grade answers    |

**Key points:**

- Evaluators are async callables returning `dict[str, float]`
- Wire them into training via `evaluator_builders` (list of zero-arg factories)
- `eval_every` controls how often they run
- Use `TrainingClientEvaluator` for metrics that only need a forward pass (fast)
- Use `SamplingClientEvaluator` for metrics that need generation (slower but more informative)
- For standard benchmarks, use the Inspect AI integration