Findings
Field notes from the bench — what the runs actually taught us. Every claim links to the attempt transcripts and reports that back it.
The open loop — a feedback loop is useless if you never close it
Every sub-10B model scored worse WITH tools than without. The transcripts show why — two of them went 0 for 18 sessions on ever running the tests.
Perfect, when it answers — the 100.0 with an asterisk
qwen3:30b-a3b graded 100 on every attempt it delivered — and never delivered a third of them. For thinking models, latency is a failure mode.
The anatomy of a 48.6 — what partial credit really means
A calculator that can't calculate still scores 48.6, because it shipped the right file with the right function names. Read the axes, not the blend.
Knowing when to stop is a separate skill
Models solved the task on turn 1, then burned the other 23 turns rewriting the same file — completion and termination are different capabilities.
The JSON-escaping cliff
Small models can write the code but can't ship it — the JSON action protocol collapses the moment the payload contains quotes.