Het’s Lab - Three Runnables and a Benchmark Proposal
My own additions, clearly marked: three Python labs and WSP-Bench-Lite.
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
Everything in this chapter is mine (Het’s Lab), not the authors’: three tiny runnable labs that make the paper’s mechanisms tangible, and a community benchmark proposal - WSP-Bench-Lite - designed to close exactly the gaps the independent review found.
| Track | Task type | Mandatory baselines | Metrics |
|---|---|---|---|
| READ | Identify behaviour from weights | Human interpretability researcher; the same heuristic written as ordinary code | Accuracy, calibration, false-positive rate |
| FIX | Repair a seeded defect | Decoding tweaks; standard direction ablation; random-direction edit control | Success rate, edit sparsity s, functional drift D |
| WRITE | Produce a working policy from a spec | Random init + short training; the best published checkpoint | Score, forward-pass budget curve fidelity(n), cost |
{
"task_id": "FIX-kine-ramp-01",
"track": "FIX",
"author_team": "alpha", // NEVER the evaluated team
"target": {
"family": "mlp", // mlp | transformer | ddpm_head
"shape": [679, 128, 640, 16],
"init": "ppo_trained",
"defect": "seeded: ramp_thruster_sign_flip"
},
"spec": "The car fails the ramp. Fix it.",
"budget": { "forward_passes": 10, "reasoning_tokens": null },
"baselines_required": ["random_direction_ablation",
"standard_direction_ablation",
"decoding_tweak", "human_interpreter"],
"metrics": ["success_rate", "edit_sparsity_s", "functional_drift_D",
"budget_curve_fidelity_n", "cost_usd"],
"seeds": 5,
"report": { "pre_registered_prompt": "sha256:...", "public_logs": true }
}Lab 01 - compile a rule. A complete paddle-following network written as numbers, by hand, with the asserts passing: run it and you have a Breakout-ish brain in 17 numbers. It makes the paper’s “reason → program → compile” mechanism concrete - you are the reasoner this time. Lab 02 - direction surgery. Ten lines that remove a direction u from a random matrix: the printout shows uᵀW′ ≈ 1e-16 after surgery, and shows the sparsity/energy trade-off when you only edit the top-k columns - the exact operation class of Experiment 8.
Lab 03 - blast radius. The paper’s arithmetic: 131,072 edited out of 0.8B = 0.016%. Run it and see where the 131,072 comes from (2 layers × 64 columns × 1024 - hidden size inferred, flagged for verification), and how tiny the fraction really is. Then WSP-Bench-Lite: three tracks (Read/Fix/Write), tasks authored by non-competing teams, ≥5 seeds, mandatory baselines (including “the same trick written as ordinary code”), metrics that include edit sparsity and functional drift, pre-registered prompts, public logs.
Why the boring rules matter: every one of them exists because its absence is how the review deflated a claim. Author-blind tasks kill the “author-designed” confound; seeds kill n=1; baselines kill “works” without “works better than a 20-line script”; drift D catches edits that fix the symptom and break the neighbour. The benchmark is the review, turned into a protocol.
Foundational Refreshers (1)