What is this?
This is mostly a vibe-coded benchmark report I've made to test out large language models that I am able to run locally, in an effort to see if any one of them will be able to sustain my personal assistant agent project – hex.
The models are being tested on the following hardware:
- CPU:
AMD Ryzen 5600x - Motherboard:
Asus B650M Aorus Elite AX ICE - Memory:
2 x 16GB DDR5 - GPU:
Gigabyte Radeon RX6750XT Gaming OC 12G (gfx1031) VulkanMemory OC @ 2312MHz - OS:
CachyOS June 2026 - Inference:
llama.cpp mainlinerunningllama serve -ngl 99 -c 60000 -ctk q4_0 -ctv q4_0 -b 512 -ub 512 --jinja -fa on
Why did scores jump? Scoring scheme v6 (2026-08-29) fixed the grader: it now sees the model's tool calls and results instead of only the final answer, so grounding it actually did — but that the old blind judge couldn't verify — no longer costs points (tool_use, verbosity). Every run in this report was re-scored under v6, so all numbers are comparable to each other — but not to screenshots of older versions of this page.
What this measures
hex-bench is a personal benchmark for the hex project that tests the personal-assistant workflow of this project: every task goes through Inspect AI’s react solver, which may call real tools — Todoist, Obsidian, web search — against an OpenAI-compatible endpoint: a local llama.cpp server, or a cloud API for the baselines. Each of the six categories below has three tasks (18 in the full suite); every task is scored per epoch.
cloud-baseline runs are made against bigger cloud models to establish a point of comparison of sorts.
- Tool use — calls the right real tool (Todoist / Obsidian / web) on live data.
- Reasoning — multi-step analysis over retrieved information.
- Verbosity — answer length: complete but no filler.
- Agentic loop — knowing when to stop acting and answer.
- Understanding — answers that match the actual source material.
- Response format — output conforms to the required format (bullets, JSON, YAML, table, …).
score = mean of all scorer values for the task (0–1).
quality = mean of the rubric scorers: judge/gated_judge, loop_eval, clarification_eval.
protocol = mean of the deterministic compliance scorers: tool_used, tool_validity, fmt_*.
pass score ≥ 1.0 · partial 0 < score < 1 · fail score = 0.
Timings — eval t/s, gen t/s, and wall time come from sample_speed(); older runs may lack them (shown as –).
± / green band — 95% uncertainty on the pass rate at this sample count (Wilson interval). Overlapping bands mean the models are statistical ties, not a real ranking.