# tinker_cookbook.stores.EvalStore

## *class* [**tinker_cookbook.stores.EvalStore**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L71)()

Manages evaluation runs across checkpoints.

All file I/O goes through the [`Storage`](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/stores/storage/index.md) protocol, making this backend-agnostic (local disk, S3, GCS).

Pickle-serializable when freshly constructed.

### [**url**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L91)(*path*)

Return a human-readable URI for a path within this eval store.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L91) (*str*)

**Returns:** *str*

### [**create_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L134)(*model_name*, *benchmarks*, *checkpoint_path*, *checkpoint_name*, *config*, *run_id*)

Create a new evaluation run and return its run_id.

**Parameters:**

- [**model_name**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L136) (*str*)
- [**benchmarks**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L137) (*list[str]*)
- [**checkpoint_path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L138) (*str | None*)
- [**checkpoint_name**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L139) (*str | None*)
- [**config**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L140) (*dict | None*)
- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L141) (*str | None*)

**Returns:** *str*

### [**run_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L174)(*run_id*)

Return filesystem path for backward compat with BenchmarkConfig.save_dir.

Only works with LocalStorage (returns a local path string). For cloud backends, use `url()` on the storage directly.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L174) (*str*)

**Returns:** *str*

### [**finalize_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L188)(*run_id*)

Collect scores from benchmark results and update metadata.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L188) (*str*)

**Returns:** *RunMetadata*

### [**list_runs**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L206)()

List all evaluation runs, most recent first.

**Returns:** *list[RunMetadata]*

### [**read_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L218)(*run_id*)

Load metadata for a specific run. Raises FileNotFoundError if missing.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L218) (*str*)

**Returns:** *RunMetadata*

### [**list_benchmarks**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L225)(*run_id*)

List benchmark names that have results for a run.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L225) (*str*)

**Returns:** *list[str]*

### [**read_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234)(*run_id*, *benchmark*)

Get aggregated result for a benchmark.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234) (*str*)

**Returns:** *BenchmarkResult | None*

### [**read_trajectories**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L248)(*run_id*, *benchmark*, *correct_only*, *incorrect_only*, *errors_only*)

Get trajectories with optional filtering.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L250) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L251) (*str*)
- [**correct_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L253) (*bool*)
- [**incorrect_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L254) (*bool*)
- [**errors_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L255) (*bool*)

**Returns:** *list[StoredTrajectory]*

### [**read_single_trajectory**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L276)(*run_id*, *benchmark*, *idx*)

Get a single trajectory by index (O(n) scan — loads all trajectories).

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) (*str*)
- [**idx**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) (*int*)

**Returns:** *StoredTrajectory | None*

### [**read_summary**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L285)(*run_id*)

Read the combined summary for a run, or `None` if missing.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L285) (*str*)

**Returns:** *dict[str, Any] | None*

### [**write_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291)(*run_id*, *result*)

Save a benchmark result.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291) (*str*)
- [**result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291) (*BenchmarkResult*)

**Returns:** *None*

### [**write_trajectory**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298)(*run_id*, *benchmark*, *traj*)

Append one trajectory to the JSONL file.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) (*str*)
- [**traj**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) (*StoredTrajectory*)

**Returns:** *None*

### [**write_summary**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302)(*run_id*, *results*)

Save a combined summary.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302) (*str*)
- [**results**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302) (*dict[str, BenchmarkResult]*)

**Returns:** *None*

### [**delete_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L314)(*run_id*)

Delete all data for a run. Idempotent (no error if already gone).

Removes metadata, summary, and all benchmark result/trajectory files. The `runs.jsonl` index is append-only and not modified; `list_runs()` checks for `metadata.json` existence so deleted runs are excluded.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L314) (*str*)

**Returns:** *None*

### [**alist_runs**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L331)()

Async version of `list_runs`.

**Returns:** *list[RunMetadata]*

### [**aread_trajectories**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L335)(*run_id*, *benchmark*, \*\**kw*)

Async version of `read_trajectories`.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) (*str*)
- [**\*\*kw**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) (*Any*)

**Returns:** *list[StoredTrajectory]*

### [**aread_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341)(*run_id*, *benchmark*)

Async version of `read_result`.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341) (*str*)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341) (*str*)

**Returns:** *BenchmarkResult | None*