INDEPENDENT RESEARCH MONOGRAPH

Weight-Space
Programming

A general-purpose LLM can inspect, repair, and synthesize raw neural-network weights through symbolic reasoning alone-completely bypassing gradient descent. An empirical breakdown from first principles to frontier replication.

3D θ-SPACE · LAYER 15 · MLP W_gate

Inspect raw weights. No forward pass, no tools.

MATH-500
7.9%
Token-limit hits
93.0%
Params edited
0 / 800M
hover the matrix to read a weightTeaching schematic · figures paper-reported (Exp 8)
Architectural ShiftFrom Black-Box SGD to Parameter Surgery

Treating Neural Network Weights as Human-Readable, Modifiable Source Code

Bypassing gradient backpropagation by projecting out defective reasoning directions: W' = (I - u·uᵀ)W modifies only 0.016% of weights to unlock latent capabilities without retraining.

Adaptive Experience Pathways

Choose How You Want to Explore Weight-Space Programming

Select your background to instantly tailor the curriculum rigor, highlight relevant tooling, and reveal curated starting points.

Active Rigor: Level 2 of 5
Curated Focus for Curious Observer & Executive

Understand what it means for an AI to inspect and fix neural network weights directly without training runs, why this alters compute economics, and what risks matter.

Interactive Parameter Simulator · Exp 8

Perform Rank-1 Surgery on Qwen-0.8B

Drag the surgery projection slider to adjust W' = (I - α·uuᵀ)W across 64 MLP columns in Layer 15 without running backpropagation.

Projection Coefficient α:1.00
0.0 (Unpatched / Runaway Loop)0.5 (Partial Damping)1.0 (Full Singular Orthogonalization)
MATH-500 ACCURACY48.1%+40.2% Gain
RUNAWAY THINKING33%-60% Stalls
EDIT SPARSITY s0.016%1 Knob in 6,000
PARAMETERS EDITED131,072 / 800MZero Retraining
Layer 15 MLP Matrix Columns (0..63)Orthogonal Cut
Excised Reasoning StallFull Derivation
Prologue Spotlight · Exp 1 In Slow MotionEmpirical Data

The Atari Weight Fingerprints

Takehi et al. gave an LLM four policies with 17,253 identical weights and asked it to name each game with zero tools. Explore the real weights recomputed live from the authors’ published GitHub prompts.

Explore The Story So Far
Paper-reportedExp 1 · Recomputed Raw Weights

The Fingerprint Lab: Reading Atari Policies

An LLM was handed four lists of 17,253 numbers with zero labels or tools and asked: Which game is which? Recomputed in real-time from the authors’ published GitHub prompts.

Breakout Architecture & Task

PPO 1B Steps

Ball bouncing against breakable brick rows above a horizontal paddle.

Mechanistic Fingerprint:

Row 8 accounts for 57.7% of all Agent channel first-layer weight energy. The paddle is constrained to move along a single horizontal axis at the bottom.

Input Observation Channel
Tensor Geometry & L2 FormulationW₁ ∈ ℝ³²ˣ⁵⁰⁰

The policy network receives an input observation tensor X ∈ ℝ¹⁰ˣ¹⁰ˣ⁵ (500 features across 5 channels). The sensitivity of spatial coordinate (r, c) is computed via the L2 norm across all 32 first-layer hidden units:

‖W₁[·, r, c, ch]‖₂ = √( Σₖ₌₁³² W₁[k, r, c, ch]² )

An LLM inspecting raw weights evaluates these marginal sums symbolically to distinguish vertical boundary constraints (Pong) from horizontal paddle tracking (Breakout) with zero forward execution passes.

10 × 10 Spatial Input Grid: L2 Norm of First-Layer Weights500 Inputs × 32 Hidden Units
0
5%
1
5%
2
4%
3
5%
4
5%
5
5%
6
5%
7
4%
8
58%
9
5%
4%
0
15%
1
11%
2
9%
3
10%
4
8%
5
9%
6
12%
7
17%
8
5%
9
Hover any grid cell to inspect exact L2 weight norm
Energy DensityMarginal Peak
Replication & Audit Log

The Living Research Ledger

Interactive timeline tracing discovery from first-principles static reading through laptop replication, 0.016% surgical repair, and academic auditing.

Full Audit Protocol
Research Progression & Methodology TimelineTracing discovery from first-principles static reading to verifiable benchmarks
3 / 5
[REC-03]
Paper-reported4 Nov 2026 · Stage 03

The 0.016% Surgical Direction Removal (Exp 8)

Zero-gradient repair of runaway reasoning loops in Qwen3.5-0.8B

Authors:Takehi et al.
Affiliation:MIT CSAIL
Core Research Question

“Can an LLM diagnose and physically excise a pathological thought loop from weights?”

Empirical Honesty & Limitations

The strongest result in the literature. Crucially, the negative control on Gemma 3 yielded 0% improvement, proving the surgery removed a real obstacle rather than injecting magical ability.

View Chapter 5 Surgery Protocol
python lab/02_direction_surgery.py
Methodological Procedure
  1. 01.Identified looping vector u in residual stream re-injected by MLPs of layers 15 and 19.
  2. 02.Computed projection operator: W' = (I - u u^T)W on 64 output columns per layer.
  3. 03.Modified exactly 131,072 parameters out of 800,000,000 (0.016% total blast radius).
  4. 04.Executed negative control edit: perturbed identical parameter count in random direction.
Measured Empirical Deliverables
MATH-500 Accuracy+40.2%
7.9%48.1%
At 50 symbolic iteration passes
GSM8K Arithmetic+53.3%
13.8%67.1%
Substantial benchmark restoration
Token-Limit Exceeded Hits-59.9%
93.0%33.1%
Direct mechanical proof that loop was broken
Continuous peer ledger. Methodological updates logged in order of empirical release.
Status: Verified AttributionAudit Protocol
Zero Hallucination Protocol

The Truth Contract

Every single metric, parameter table, and schematic diagram carries exactly one provenance badge. Press T anywhere on this guide to strip away speculation and toy schematics, leaving only empirical paper facts and independent replications.

Why this matters: AI research has an epidemic of narrative exaggeration. This site strictly distinguishes between claims verified by formal benchmarks versus pedagogical toy illustrations.

Toggle Filter: TShortcut Guide: ?

Provenance Badging Hierarchy

  • Paper-reportedA number or claim stated in the original post (2 Nov 2026).
  • Independent reviewFrom a third-party analysis (e.g. Maier, 5 Nov 2026).
  • Teaching schematicMade-up toy data created to explain an idea. Illustrative, not real data.
  • SpeculationPredictions about the future - untested, including the authors’ own.
Flagship Empirical RecordPaper-reported

The 0.016% Surgery (Exp 8)

Modifying just 131,072 parameters out of 800 million in Qwen3.5-0.8B restored reasoning accuracy on MATH-500 from 7.9% to 48.1%.

Negative control edit (perturbed same size in arbitrary direction) yielded zero gain, confirming surgical precision.

Takehi et al. · CSAILRead Surgery Breakdown

Field Guide Curriculum

Four parts spanning 13 canonical chapters. Every paragraph can be read across 5 audience rungs-from intuitive conceptual metaphors to frontier mathematical rigor.

Research Instruments

Directly interact with empirical material through runnable Python scripts, cognitive quizzes, and verified citation ledgers.

Runnable Python Labs

Three self-contained scripts: hand-compile a policy, perform direction surgery, and compute blast radii with zero external dependencies.

Open Lab Scripts

Cognitive Mastery Quiz

Fifteen challenging questions testing your grasp from weight representations to ablation design and empirical critiques.

Take the Quiz

Technical FAQs

12 searchable answers: zero-gradient hardware needs, LoRA vs WSP, Qwen 0.8B direction surgery, and replication bounds.

Explore FAQs

Whitepaper

Complete scholarly PDF with every derivation, parameter tensor matrices, full bibliography, and mathematical proofs.

Download PDF

The Honesty Ledger

44 bibliography references tagged by verification tier, plus the complete catalogue of unconfirmed research bounds.

Audit the Ledger
Open Curriculum · 13 Chapters

Ready to dive inside the weights?

Explore the complete field guide from first principles to zero-gradient directional surgery.