Part 3·Chapter 8 of 13· 9 min read

Measured vs Anecdote - Sorting the Evidence

A scoreboard, a review, and the difference between sparks and fireworks.

Reading Audience LevelRung 2: L2 · Curious adult

Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.

A third-party review (Andreas Maier, 5 Nov 2026) sorted every claim in the paper by evidence strength. This chapter walks that scoreboard, names the missing baselines, and explains why “sparks” is exactly the right word.

Independent reviewEvidence vs anecdote - the sorted scoreboard
Evidence strengthWhatWhy
StrongExp 8 (Qwen math fix)Measured on 3 benchmarks; mechanism checked via token-limit share; negative control included
StrongExp 4 (ramp fix)Large effect, measured success rate, one-shot
Strong (with n=1 caveat)Exp 5 & 6 (writing from a designed program)Working policies from zero - but single runs
WeakerExp 1 & 2 (reads on toy nets)4 and 3 correct answers; readable layouts
WeakerExp 3 language ID~0.5 confidence over 3 options; copying-head match is the solid part
WeakerExp 7 sycophancy3 binary picks = 1-in-8 by chance; task had to be relaxed
Missing everywhereBaselines & seedsOther LLMs, human/automated baselines, ≥5 seeds, cheaper alternatives, author-blind tasks
Independent review

M14 · Evidence vs Hype Scoreboard

m14-scoreboard
Evidentiary Audit: Exp 8: Multi-benchmark validation, verified token-limit mechanism, and a strict negative control demonstrating targeted excision.

Calibration of prominent claims against Maier's independent review (🟩). Click any claim to inspect the evidentiary rationale. Note: Scores are qualitative editorial indicators, not meta-analytic absolutes.

L2 view: Green bars represent careful empirical measurements with controls; blue indicates solid demonstrations; amber highlights anecdotal findings; red marks untested speculation.

🟩 Qualitative assessment calibrated against Maier's independent review (5 Nov 2026).

Paper-reported

M6 · The Budget Dial

m6-budget-dial
7.9
0 passes
12.3
10 passes
14.1
25 passes
48.1
50 passes

Qwen3.5-0.8B · thinking mode on · up to 32,768 tokens - one measurement per bar, reported points only

Qwen3.5-0.8B, MATH-500 %: 0→7.9 · 10→12.3 · 25→14.1 · 50→48.1. the cliff: 14.1 at 25 passes → 48.1 at 50 - a jump, not a slope Reported points only - no curve drawn through them.

L2 view: Each bar is one measured score. The jump at 50 tries is huge (7.9% → 48.1% on MATH-500) - but each bar is a single run, so treat wiggles gently.

🟦 All values verbatim from the paper’s tables (Exp 8). Chart: reported points as bars, focus ring controlled by Play/Step.

Independent review

“the strongest evidence is the Qwen math edit - measured, mechanism-checked, controlled”

- Andreas Maier, “What is measured and what is anecdote?”, 5 Nov 2026 (🟩, paraphrased summary)

The sorting, from the review (🟩). Strongest first: Exp 8 (the Qwen math fix - measured on MATH-500/GSM8K/IFEval, mechanism checked via the token-limit share, with the Gemma negative control). Then Exp 4 (the ramp fix: 0/256 → 1,024/1,024, one shot). Then Exps 5–6 (writing working policies from designed programs - real artifacts beating real baselines, but each a single run).

Weaker, per the same review: Exps 1–2 (four and three correct answers on tiny, readable toys), Exp 3’s language ID (confidence 0.48–0.62 among three options - barely better than a die with three sides; the copying-head match is the solid part), and Exp 7 (three binary picks, a 1-in-8 chance of pure luck, and the task had to be relaxed before it worked).

Missing everywhere: other LLMs (does this work beyond one frontier model?), human or automated baselines, repeated runs with different seeds, cheap alternatives (decoding tweaks, standard direction ablation, the same heuristic as ordinary code), and - most structurally important - tasks not designed by the authors.

Foundational Refreshers (1)