Part 4·Chapter 12 of 13· 9 min read

Het’s Lab - Three Runnables and a Benchmark Proposal

My own additions, clearly marked: three Python labs and WSP-Bench-Lite.

Reading Audience LevelRung 2: L2 · Curious adult

Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.

Everything in this chapter is mine (Het’s Lab), not the authors’: three tiny runnable labs that make the paper’s mechanisms tangible, and a community benchmark proposal - WSP-Bench-Lite - designed to close exactly the gaps the independent review found.

SpeculationWSP-Bench-Lite - proposed scoring template (one page)
TrackTask typeMandatory baselinesMetrics
READIdentify behaviour from weightsHuman interpretability researcher; the same heuristic written as ordinary codeAccuracy, calibration, false-positive rate
FIXRepair a seeded defectDecoding tweaks; standard direction ablation; random-direction edit controlSuccess rate, edit sparsity s, functional drift D
WRITEProduce a working policy from a specRandom init + short training; the best published checkpointScore, forward-pass budget curve fidelity(n), cost
Het’s original proposal (clearly credited to Het Patel, not the authors). Tasks must not be authored by the evaluated team; ≥5 seeds; multiple LLMs; pre-registered prompts; public logs.
json
{
  "task_id": "FIX-kine-ramp-01",
  "track": "FIX",
  "author_team": "alpha",          // NEVER the evaluated team
  "target": {
    "family": "mlp",               // mlp | transformer | ddpm_head
    "shape": [679, 128, 640, 16],
    "init": "ppo_trained",
    "defect": "seeded: ramp_thruster_sign_flip"
  },
  "spec": "The car fails the ramp. Fix it.",
  "budget": { "forward_passes": 10, "reasoning_tokens": null },
  "baselines_required": ["random_direction_ablation",
                          "standard_direction_ablation",
                          "decoding_tweak", "human_interpreter"],
  "metrics": ["success_rate", "edit_sparsity_s", "functional_drift_D",
              "budget_curve_fidelity_n", "cost_usd"],
  "seeds": 5,
  "report": { "pre_registered_prompt": "sha256:...", "public_logs": true }
}
WSP-Bench-Lite - JSON task schema (draft proposal).

Lab 01 - compile a rule. A complete paddle-following network written as numbers, by hand, with the asserts passing: run it and you have a Breakout-ish brain in 17 numbers. It makes the paper’s “reason → program → compile” mechanism concrete - you are the reasoner this time. Lab 02 - direction surgery. Ten lines that remove a direction u from a random matrix: the printout shows uᵀW′ ≈ 1e-16 after surgery, and shows the sparsity/energy trade-off when you only edit the top-k columns - the exact operation class of Experiment 8.

Lab 03 - blast radius. The paper’s arithmetic: 131,072 edited out of 0.8B = 0.016%. Run it and see where the 131,072 comes from (2 layers × 64 columns × 1024 - hidden size inferred, flagged for verification), and how tiny the fraction really is. Then WSP-Bench-Lite: three tracks (Read/Fix/Write), tasks authored by non-competing teams, ≥5 seeds, mandatory baselines (including “the same trick written as ordinary code”), metrics that include edit sparsity and functional drift, pre-registered prompts, public logs.

Why the boring rules matter: every one of them exists because its absence is how the review deflated a claim. Author-blind tasks kill the “author-designed” confound; seeds kill n=1; baselines kill “works” without “works better than a 20-line script”; drift D catches edits that fix the symptom and break the neighbour. The benchmark is the review, turned into a protocol.

Foundational Refreshers (1)