Measured vs Anecdote - Sorting the Evidence
A scoreboard, a review, and the difference between sparks and fireworks.
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
A third-party review (Andreas Maier, 5 Nov 2026) sorted every claim in the paper by evidence strength. This chapter walks that scoreboard, names the missing baselines, and explains why “sparks” is exactly the right word.
| Evidence strength | What | Why |
|---|---|---|
| Strong | Exp 8 (Qwen math fix) | Measured on 3 benchmarks; mechanism checked via token-limit share; negative control included |
| Strong | Exp 4 (ramp fix) | Large effect, measured success rate, one-shot |
| Strong (with n=1 caveat) | Exp 5 & 6 (writing from a designed program) | Working policies from zero - but single runs |
| Weaker | Exp 1 & 2 (reads on toy nets) | 4 and 3 correct answers; readable layouts |
| Weaker | Exp 3 language ID | ~0.5 confidence over 3 options; copying-head match is the solid part |
| Weaker | Exp 7 sycophancy | 3 binary picks = 1-in-8 by chance; task had to be relaxed |
| Missing everywhere | Baselines & seeds | Other LLMs, human/automated baselines, ≥5 seeds, cheaper alternatives, author-blind tasks |
Calibration of prominent claims against Maier's independent review (🟩). Click any claim to inspect the evidentiary rationale. Note: Scores are qualitative editorial indicators, not meta-analytic absolutes.
L2 view: Green bars represent careful empirical measurements with controls; blue indicates solid demonstrations; amber highlights anecdotal findings; red marks untested speculation.
🟩 Qualitative assessment calibrated against Maier's independent review (5 Nov 2026).
Qwen3.5-0.8B · thinking mode on · up to 32,768 tokens - one measurement per bar, reported points only
Qwen3.5-0.8B, MATH-500 %: 0→7.9 · 10→12.3 · 25→14.1 · 50→48.1. the cliff: 14.1 at 25 passes → 48.1 at 50 - a jump, not a slope Reported points only - no curve drawn through them.
L2 view: Each bar is one measured score. The jump at 50 tries is huge (7.9% → 48.1% on MATH-500) - but each bar is a single run, so treat wiggles gently.
🟦 All values verbatim from the paper’s tables (Exp 8). Chart: reported points as bars, focus ring controlled by Play/Step.
Independent review“the strongest evidence is the Qwen math edit - measured, mechanism-checked, controlled”
- Andreas Maier, “What is measured and what is anecdote?”, 5 Nov 2026 (🟩, paraphrased summary)
The sorting, from the review (🟩). Strongest first: Exp 8 (the Qwen math fix - measured on MATH-500/GSM8K/IFEval, mechanism checked via the token-limit share, with the Gemma negative control). Then Exp 4 (the ramp fix: 0/256 → 1,024/1,024, one shot). Then Exps 5–6 (writing working policies from designed programs - real artifacts beating real baselines, but each a single run).
Weaker, per the same review: Exps 1–2 (four and three correct answers on tiny, readable toys), Exp 3’s language ID (confidence 0.48–0.62 among three options - barely better than a die with three sides; the copying-head match is the solid part), and Exp 7 (three binary picks, a 1-in-8 chance of pure luck, and the task had to be relaxed before it worked).
Missing everywhere: other LLMs (does this work beyond one frontier model?), human or automated baselines, repeated runs with different seeds, cheap alternatives (decoding tweaks, standard direction ablation, the same heuristic as ordinary code), and - most structurally important - tasks not designed by the authors.
Foundational Refreshers (1)