Weight-Space
Programming
A general-purpose LLM can inspect, repair, and synthesize raw neural-network weights through symbolic reasoning alone-completely bypassing gradient descent. An empirical breakdown from first principles to frontier replication.
Inspect raw weights. No forward pass, no tools.
- MATH-500
- 7.9%
- Token-limit hits
- 93.0%
- Params edited
- 0 / 800M
Treating Neural Network Weights as Human-Readable, Modifiable Source Code
Bypassing gradient backpropagation by projecting out defective reasoning directions: W' = (I - u·uᵀ)W modifies only 0.016% of weights to unlock latent capabilities without retraining.
Choose How You Want to Explore Weight-Space Programming
Select your background to instantly tailor the curriculum rigor, highlight relevant tooling, and reveal curated starting points.
Understand what it means for an AI to inspect and fix neural network weights directly without training runs, why this alters compute economics, and what risks matter.
Three Verbs That Define It
Read the Weights. Fix and Write into Weights. Audit the Artifact. Each verb carries distinct empirical verification criteria, mechanisms, and architectural bounds.
Read the Weights
Locate exact copying attention heads and identify game policies and visual diffusion targets simply by inspecting weight matrices-zero model execution required.
Demonstrated on toy MLPs and 4-layer transformers (17K–7.16M weights). Proves structured semantic information is legible to reasoning models.
Write into Weights
Apply targeted rank-1 weight surgeries (ΔW = u·vᵀ) to fix neural pathways or synthesize working parameters from scratch. A 0.016% edit lifted MATH-500 from 7.9% to 48.1%.
Strongest finding: negative control verified an architectural reasoning stall was excised, preserving general model capabilities.
Audit the Artifact
De-bias claims through academic scrutiny. Separates verified zero-gradient breakthroughs from single-run narrative extrapolation and establishes the WSP-Bench-Lite standard.
Catalogued all 8 experiments into three empirical tiers: 3 Strong, 3 Moderate, 2 Weak. Identified single-seed variance and budget curve gaps.
Perform Rank-1 Surgery on Qwen-0.8B
Drag the surgery projection slider to adjust W' = (I - α·uuᵀ)W across 64 MLP columns in Layer 15 without running backpropagation.
The Atari Weight Fingerprints
Takehi et al. gave an LLM four policies with 17,253 identical weights and asked it to name each game with zero tools. Explore the real weights recomputed live from the authors’ published GitHub prompts.
The Fingerprint Lab: Reading Atari Policies
An LLM was handed four lists of 17,253 numbers with zero labels or tools and asked: Which game is which? Recomputed in real-time from the authors’ published GitHub prompts.
Breakout Architecture & Task
PPO 1B StepsBall bouncing against breakable brick rows above a horizontal paddle.
Row 8 accounts for 57.7% of all Agent channel first-layer weight energy. The paddle is constrained to move along a single horizontal axis at the bottom.
The policy network receives an input observation tensor X ∈ ℝ¹⁰ˣ¹⁰ˣ⁵ (500 features across 5 channels). The sensitivity of spatial coordinate (r, c) is computed via the L2 norm across all 32 first-layer hidden units:
An LLM inspecting raw weights evaluates these marginal sums symbolically to distinguish vertical boundary constraints (Pong) from horizontal paddle tracking (Breakout) with zero forward execution passes.
The Living Research Ledger
Interactive timeline tracing discovery from first-principles static reading through laptop replication, 0.016% surgical repair, and academic auditing.
The 0.016% Surgical Direction Removal (Exp 8)
Zero-gradient repair of runaway reasoning loops in Qwen3.5-0.8B
“Can an LLM diagnose and physically excise a pathological thought loop from weights?”
The strongest result in the literature. Crucially, the negative control on Gemma 3 yielded 0% improvement, proving the surgery removed a real obstacle rather than injecting magical ability.
python lab/02_direction_surgery.py- 01.Identified looping vector u in residual stream re-injected by MLPs of layers 15 and 19.
- 02.Computed projection operator: W' = (I - u u^T)W on 64 output columns per layer.
- 03.Modified exactly 131,072 parameters out of 800,000,000 (0.016% total blast radius).
- 04.Executed negative control edit: perturbed identical parameter count in random direction.
The Truth Contract
Every single metric, parameter table, and schematic diagram carries exactly one provenance badge. Press T anywhere on this guide to strip away speculation and toy schematics, leaving only empirical paper facts and independent replications.
Why this matters: AI research has an epidemic of narrative exaggeration. This site strictly distinguishes between claims verified by formal benchmarks versus pedagogical toy illustrations.
Provenance Badging Hierarchy
- Paper-reportedA number or claim stated in the original post (2 Nov 2026).
- Independent reviewFrom a third-party analysis (e.g. Maier, 5 Nov 2026).
- Teaching schematicMade-up toy data created to explain an idea. Illustrative, not real data.
- SpeculationPredictions about the future - untested, including the authors’ own.
The 0.016% Surgery (Exp 8)
Modifying just 131,072 parameters out of 800 million in Qwen3.5-0.8B restored reasoning accuracy on MATH-500 from 7.9% to 48.1%.
Negative control edit (perturbed same size in arbitrary direction) yielded zero gain, confirming surgical precision.
Field Guide Curriculum
Four parts spanning 13 canonical chapters. Every paragraph can be read across 5 audience rungs-from intuitive conceptual metaphors to frontier mathematical rigor.
Part 1: Part I · Foundations
What weights are, how they get written today, and the family of cousin methods.
Part 2: Part II · The Experiments
Read, fix, write - all eight experiments, and how the reasoning actually worked.
Part 3: Part III · Judgement
Measured vs anecdote, and who benefits - the reviewer’s toolkit.
Part 4: Part IV · Futures & Risks
Predictions, the risk atlas, Het’s Lab, and the library.
Research Instruments
Directly interact with empirical material through runnable Python scripts, cognitive quizzes, and verified citation ledgers.
Runnable Python Labs
Three self-contained scripts: hand-compile a policy, perform direction surgery, and compute blast radii with zero external dependencies.
Cognitive Mastery Quiz
Fifteen challenging questions testing your grasp from weight representations to ablation design and empirical critiques.
Technical FAQs
12 searchable answers: zero-gradient hardware needs, LoRA vs WSP, Qwen 0.8B direction surgery, and replication bounds.
Whitepaper
Complete scholarly PDF with every derivation, parameter tensor matrices, full bibliography, and mathematical proofs.
The Honesty Ledger
44 bibliography references tagged by verification tier, plus the complete catalogue of unconfirmed research bounds.
Ready to dive inside the weights?
Explore the complete field guide from first principles to zero-gradient directional surgery.