Part 1·Chapter 2 of 13· 10 min read

How Weights Get Written - Three Old Ways, One New

Optimization, trained writers, compilers… and now: reasoning.

Reading Audience LevelRung 2: L2 · Curious adult

Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.

Before November 2026, weights were written in three ways: by optimization grinding downhill, by trained writer-models, or by compilers starting from formal programs. Weight-space programming proposes a fourth: a general-purpose LLM that reads θ, reasons, and writes Δ directly.

Teaching schematicFour paradigms for writing weights - ①②③ existed before; ④ is the new claim. ILLUSTRATIVE.
① Optimization

SGD · Adam · Muon - millions of tiny symbolic steps

② Trained writers

hypernetworks · fast weights · weight diffusion

③ Compilers

Tracr - formal program → weights

④ WSP

a general LLM reasons about θ and writes Δ directly

Teaching schematic

M2 · Two Ways to Move

m2-two-ways

① Optimization - many small steps

θ*

steps so far: 0

④ Weight-space programming - one reasoned jump

θθ + Δ

forward passes before the edit: 0 · Δ decided by: reasoning

Count the cost difference - that’s the whole pitch, and the whole open question.

Left: 0 tiny gradient steps so far (learning rate η = 0.35). Right: one reasoned jump, 0 forward passes before the edit.

L2 view: Today’s training nudges weights millions of times. A weight-space edit picks the destination in one go - the cost is thinking, not stepping.

TEACHING SCHEMATIC - the “hill” is invented. The step/jump contrast mirrors the paper’s method comparison, but no real loss curve looks this tidy.

Mathematical Formulations (2)Comparative Analysis
Paper-reportedTraining - how weights get written today
f-sgd

Gradient descent nudges every weight a tiny step downhill on the loss, repeated millions of times.

•
η (eta) - learning rate, the step size
•
∇L - the gradient: which direction reduces the loss
•
t - training step number

Example: PPO needed 100M–500M steps to reach 15.8–25.3 on MinAtar Breakout.

LaTeX code
\theta_{t+1}=\theta_t-\eta\,\nabla_\theta\mathcal{L}(\theta_t)
Paper-reportedWeight-space programming
f-wsp

An LLM reads the weights, decides a full edit Δ in one reasoned shot, and writes θ′ - no gradient descent.

•
θ′ - the edited weights
•
Δ (delta) - the edit, produced by reasoning
•
spec - the task description
•
feedback - optional forward passes the editor may run

Example: Exp 4: one Δ took a broken ramp-climber from 0/256 to 1,024/1,024 successes.

LaTeX code
\theta'=\theta+\Delta,\quad \Delta=\mathrm{LLM}\!\left(\theta,\ \text{spec},\ \text{optional feedback}\right)

① Optimization (SGD, Adam, Muon). The default. Compute how wrong the model is, work out which direction each parameter should move to be slightly less wrong, take a tiny step, repeat - typically millions of steps over billions of examples. It is the only method that has ever produced frontier models; it is also slow, data-hungry, and gives you no human-readable explanation of what changed.

② Trained writers. Hypernetworks (Ha et al., 2016) are networks that output other networks’ weights. Fast-weight programmers (Schlag et al., 2021) let a transformer write temporary weights as it runs. Neural network diffusion (Wang et al., 2024) generates whole weight sets from a learned distribution. All share one requirement: someone had to train the writer first, on lots of examples of networks.

③ Compilers. Tracr (Lindell et al., 2023) compiles a program written in RASP directly into transformer weights - a genuine “weights from recipe” path, but only for the formal, hand-written programs and toy architectures it supports.

④ Weight-space programming (WSP). The paper’s claim: a general-purpose LLM, given the weights (or slices of them) plus a plain-language task, produces an edit Δ by reasoning - optionally checking itself with a handful of forward passes. The writer was never trained to write weights; it learned to understand code, math, and systems, and transfers that understanding to weight space.

Foundational Refreshers (1)