EVIDENCE AUDIT · 8 EMPIRICAL EXPERIMENTS · VERIFIED PROVENANCE

The Experiments & Evidence Room

Eight canonical experiments deconstructed across three fundamental modalities: reading representations, fixing broken weights, and writing working models from scratch. Every experiment compares original paper-reported metrics against independent third-party replication audits.

Evidence Honesty Key

We separate empirical claims stated in the original CSAIL post from independent review ratings. Schematics and speculation are distinctly badged.

  • Paper-reportedA number or claim stated in the original post (2 Nov 2026).
  • Independent reviewFrom a third-party analysis (e.g. Maier, 5 Nov 2026).
  • Teaching schematicMade-up toy data created to explain an idea. Illustrative, not real data.
  • SpeculationPredictions about the future - untested, including the authors’ own.
Paper-reported

M6 · The Budget Dial

m6-budget-dial
7.9
0 passes
12.3
10 passes
14.1
25 passes
48.1
50 passes

Qwen3.5-0.8B · thinking mode on · up to 32,768 tokens - one measurement per bar, reported points only

Qwen3.5-0.8B, MATH-500 %: 0→7.9 · 10→12.3 · 25→14.1 · 50→48.1. the cliff: 14.1 at 25 passes → 48.1 at 50 - a jump, not a slope Reported points only - no curve drawn through them.

L2 view: Each bar is one measured score. The jump at 50 tries is huge (7.9% → 48.1% on MATH-500) - but each bar is a single run, so treat wiggles gently.

🟦 All values verbatim from the paper’s tables (Exp 8). Chart: reported points as bars, focus ring controlled by Play/Step.

Paper-reported

M8 · The Copying-Head Grid

m8-head-grid
Attention Heads (H0 … H5)H0H1H2H3H4H5Layers (L0 … L3)L0L1L2L3L0·H0L0·H1L0·H2L0·H3L0·H4L0·H5L1·H0L1·H1L1·H2L1·H3L1·H4L1·H5L2·H0L2·H1L2·H2L2·H3L2·H4L2·H5L3·H0L3·H1L3·H2L3·H3L3·H4L3·H5

Highlighted = heads Astra claimed are copiers, inferred strictly from weight composition.

Circuit Identification: Python

Target heads: L2·H1, L3·H1, L3·H5

Discovered by reading induction-head composition scores without running forward inference passes.

Why “composition score” finds copiers:

A copying head is two heads in series: one moves the previous token forward, the next uses it to find the repeat. Their weight matrices multiply, so their product’s norm reveals the chain - visible directly in the weights, no forward pass needed.

Lang:

What Astra READ from raw weights (Python): L2·H1, L3·H1, L3·H5 via induction-head composition scores alone.

L2 view: A “head” is one attention unit. The paper knew (by running the models) which heads copy. The reader found the same ones by reading the weights only - a perfect match.

🟦 Head positions verbatim from Exp 3 (tables in the post). Confidence on language ID: 0.48–0.62 among 3 options.

Paper-reported

M11 · The Blast-Radius Meter

m11-blast-radius
1 ppm0.01%1%

0.0164%

log scale ← 1-in-a-billion · · · · · · · · 100% → (Exp 8 sits at 0.016%, ~2 dots from the left)

Exp 8 · Qwen3.5-0.8B: 0.016% - the paper’s flagship edit 🟦

Editing 131,072 of 800,000,000 weights = 0.0164% - the “blast radius” s = ‖Δ‖₀ / P.

L2 view: Compare: a full re-train touches 100% of the numbers. The Qwen math fix touched 0.016% - about 1 in 6,000 - and changed the model’s exam score by 40 points.

🟦 preset values from the paper · 🟨 LoRA & full-FT rows are teaching comparisons. Your custom input is your own.

Research Frontiers (November 2026 Updates)

Independent Replications & Mechanistic Insights

Community Cross-Verification
Finding 1

Thinking Loop Attractor Collapse

Mechanistic inspection of the Qwen3.5-0.8B edit revealed that repetitive reasoning loops lived in a 1D attractor subspace in MLP layers 15 & 19. Projecting out direction u via W' = (I - uuᵀ)W destroyed this attractor without impairing mathematical deduction.

Token limit hit rate collapsed from 93.0% down to 33.1%Explains why editing 0.016% of weights unlocked latent knowledge already present in the pre-trained model.
Finding 2

Independent 3rd-Party Replications (Nov 2026)

Community evaluations on Hugging Face verified the surgical weight edit on independent MATH-500 runs, confirming scores between 48.1% and 51.2%, with zero training passes and zero optimizer states.

MATH-500 verified at 48.1%–51.2% (vs 7.9% base)Rules out benchmark evaluation artifacts in the original CSAIL demo.
Finding 3

Negative Control Integrity (Gemma-3-270M)

When the same rank-1 projection was applied to Gemma-3-270M (which has only an 8.5% token limit rate), accuracy remained flat at 6.8%–7.7%.

Gemma MATH-500: 6.8% base → 7.0% (50 passes)Demonstrates that WSP edits remove pathological obstacles rather than injecting generic reasoning abilities.
Finding 4

WSP vs Gradient Baselines (The Reachability Bound)

Theoretical comparison between symbolic weight surgery, ROME/MEMIT key-value editing, and gradient descent. WSP requires zero loss formulation or backward pass, but is currently bounded by context window capacity and parameter observability.

0 gradients · 0 backprop · 131,072 params editedEstablishes WSP as an independent fourth paradigm alongside Prompting, Fine-Tuning, and Activation Steering.

The Four Control Paradigms Compared

ParadigmOptimization MethodData / Forward BudgetWeight PermanenceInference OverheadPrimary Failure Mode
Weight-Space Programming (WSP)Symbolic deduction over raw tensors0 data · 0–50 forward passesPermanent (static delta)0% (identical architecture)Context capacity & representation undecidability
Gradient Descent / SFTBackpropagation & SGD/AdamThousands to millions of examplesPermanent0%Catastrophic forgetting & compute cost
Activation Steering / RepEngDynamic activation addition during forward passContrastive pair extraction setTransient (runtime injection)+VRAM & hook latencyInference drift & hook fragility
Model Editing (ROME / MEMIT)Closed-form least-squares key-value insertionExplicit factual triples (s, r, o)Permanent (rank-1 delta)0%Locality leakage & inability to edit reasoning
Compilation (Tracr / RASP)Deterministic compiler mapping DSL to weightsFormal algorithm specificationPermanent0%Narrow DSL expressivity; cannot edit pre-trained nets
Independent review

M14 · Evidence vs Hype Scoreboard

m14-scoreboard
Evidentiary Audit: Exp 8: Multi-benchmark validation, verified token-limit mechanism, and a strict negative control demonstrating targeted excision.

Calibration of prominent claims against Maier's independent review (🟩). Click any claim to inspect the evidentiary rationale. Note: Scores are qualitative editorial indicators, not meta-analytic absolutes.

L2 view: Green bars represent careful empirical measurements with controls; blue indicates solid demonstrations; amber highlights anecdotal findings; red marks untested speculation.

🟩 Qualitative assessment calibrated against Maier's independent review (5 Nov 2026).