The Experiments & Evidence Room
Eight canonical experiments deconstructed across three fundamental modalities: reading representations, fixing broken weights, and writing working models from scratch. Every experiment compares original paper-reported metrics against independent third-party replication audits.
Evidence Honesty Key
We separate empirical claims stated in the original CSAIL post from independent review ratings. Schematics and speculation are distinctly badged.
- Paper-reportedA number or claim stated in the original post (2 Nov 2026).
- Independent reviewFrom a third-party analysis (e.g. Maier, 5 Nov 2026).
- Teaching schematicMade-up toy data created to explain an idea. Illustrative, not real data.
- SpeculationPredictions about the future - untested, including the authors’ own.
8 experiments shown. Click any card to expand its experimental setup, measured delta, and independent review audit.
L2 view: Each card shows the target model, the editor's intervention, and the outcome. The colored evidence tags distinguish empirical measurements from anecdotal demos.
🟦 Setups & results from CSAIL paper · 🟩 Evidence assessments from Maier's independent review (5 Nov 2026).
Qwen3.5-0.8B · thinking mode on · up to 32,768 tokens - one measurement per bar, reported points only
Qwen3.5-0.8B, MATH-500 %: 0→7.9 · 10→12.3 · 25→14.1 · 50→48.1. the cliff: 14.1 at 25 passes → 48.1 at 50 - a jump, not a slope Reported points only - no curve drawn through them.
L2 view: Each bar is one measured score. The jump at 50 tries is huge (7.9% → 48.1% on MATH-500) - but each bar is a single run, so treat wiggles gently.
🟦 All values verbatim from the paper’s tables (Exp 8). Chart: reported points as bars, focus ring controlled by Play/Step.
Highlighted = heads Astra claimed are copiers, inferred strictly from weight composition.
Target heads: L2·H1, L3·H1, L3·H5
Discovered by reading induction-head composition scores without running forward inference passes.
A copying head is two heads in series: one moves the previous token forward, the next uses it to find the repeat. Their weight matrices multiply, so their product’s norm reveals the chain - visible directly in the weights, no forward pass needed.
What Astra READ from raw weights (Python): L2·H1, L3·H1, L3·H5 via induction-head composition scores alone.
L2 view: A “head” is one attention unit. The paper knew (by running the models) which heads copy. The reader found the same ones by reading the weights only - a perfect match.
🟦 Head positions verbatim from Exp 3 (tables in the post). Confidence on language ID: 0.48–0.62 among 3 options.
0.0164%
log scale ← 1-in-a-billion · · · · · · · · 100% → (Exp 8 sits at 0.016%, ~2 dots from the left)
Exp 8 · Qwen3.5-0.8B: 0.016% - the paper’s flagship edit 🟦
Editing 131,072 of 800,000,000 weights = 0.0164% - the “blast radius” s = ‖Δ‖₀ / P.
L2 view: Compare: a full re-train touches 100% of the numbers. The Qwen math fix touched 0.016% - about 1 in 6,000 - and changed the model’s exam score by 40 points.
🟦 preset values from the paper · 🟨 LoRA & full-FT rows are teaching comparisons. Your custom input is your own.
Independent Replications & Mechanistic Insights
Thinking Loop Attractor Collapse
Mechanistic inspection of the Qwen3.5-0.8B edit revealed that repetitive reasoning loops lived in a 1D attractor subspace in MLP layers 15 & 19. Projecting out direction u via W' = (I - uuᵀ)W destroyed this attractor without impairing mathematical deduction.
Independent 3rd-Party Replications (Nov 2026)
Community evaluations on Hugging Face verified the surgical weight edit on independent MATH-500 runs, confirming scores between 48.1% and 51.2%, with zero training passes and zero optimizer states.
Negative Control Integrity (Gemma-3-270M)
When the same rank-1 projection was applied to Gemma-3-270M (which has only an 8.5% token limit rate), accuracy remained flat at 6.8%–7.7%.
WSP vs Gradient Baselines (The Reachability Bound)
Theoretical comparison between symbolic weight surgery, ROME/MEMIT key-value editing, and gradient descent. WSP requires zero loss formulation or backward pass, but is currently bounded by context window capacity and parameter observability.
The Four Control Paradigms Compared
| Paradigm | Optimization Method | Data / Forward Budget | Weight Permanence | Inference Overhead | Primary Failure Mode |
|---|---|---|---|---|---|
| Weight-Space Programming (WSP) | Symbolic deduction over raw tensors | 0 data · 0–50 forward passes | Permanent (static delta) | 0% (identical architecture) | Context capacity & representation undecidability |
| Gradient Descent / SFT | Backpropagation & SGD/Adam | Thousands to millions of examples | Permanent | 0% | Catastrophic forgetting & compute cost |
| Activation Steering / RepEng | Dynamic activation addition during forward pass | Contrastive pair extraction set | Transient (runtime injection) | +VRAM & hook latency | Inference drift & hook fragility |
| Model Editing (ROME / MEMIT) | Closed-form least-squares key-value insertion | Explicit factual triples (s, r, o) | Permanent (rank-1 delta) | 0% | Locality leakage & inability to edit reasoning |
| Compilation (Tracr / RASP) | Deterministic compiler mapping DSL to weights | Formal algorithm specification | Permanent | 0% | Narrow DSL expressivity; cannot edit pre-trained nets |
Calibration of prominent claims against Maier's independent review (🟩). Click any claim to inspect the evidentiary rationale. Note: Scores are qualitative editorial indicators, not meta-analytic absolutes.
L2 view: Green bars represent careful empirical measurements with controls; blue indicates solid demonstrations; amber highlights anecdotal findings; red marks untested speculation.
🟩 Qualitative assessment calibrated against Maier's independent review (5 Nov 2026).