What this measures
hex-bench is a personal benchmark for the hex project that tests the personal-assistant workflow of this project: every task goes through Inspect AI’s react solver, which may call real tools — Todoist, Obsidian, web search — against a local OpenAI-compatible llama.cpp endpoint. Each of the six categories below has three tasks (18 in the full suite); every task is scored per epoch.
baseline runs are made against bigger cloud models to establish a point of comparison of sorts.
- Tool use — calls the right real tool (Todoist / Obsidian / web) on live data.
- Reasoning — multi-step analysis over retrieved information.
- Verbosity — answer length: complete but no filler.
- Agentic loop — knowing when to stop acting and answer.
- Understanding — answers that match the actual source material.
- Response format — output conforms to the required format (bullets, JSON, YAML, table, …).
score = mean of all scorer values for the task (0–1).
quality = mean of the rubric scorers: judge, loop_eval, clarification_eval.
protocol = mean of the deterministic compliance scorers: tool_used, fmt_*.
pass score ≥ 1.0 · partial 0 < score < 1 · fail score = 0.
Timings — eval t/s, gen t/s, and wall time come from sample_speed(); older runs may lack them (shown as –).
± / green band — 95% uncertainty on the pass rate at this sample count (Wilson interval). Overlapping bands mean the models are statistical ties, not a real ranking.