How Weights Get Written - Three Old Ways, One New
Optimization, trained writers, compilers… and now: reasoning.
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
Before November 2026, weights were written in three ways: by optimization grinding downhill, by trained writer-models, or by compilers starting from formal programs. Weight-space programming proposes a fourth: a general-purpose LLM that reads θ, reasons, and writes Δ directly.
SGD · Adam · Muon - millions of tiny symbolic steps
hypernetworks · fast weights · weight diffusion
Tracr - formal program → weights
a general LLM reasons about θ and writes Δ directly
① Optimization - many small steps
steps so far: 0
④ Weight-space programming - one reasoned jump
forward passes before the edit: 0 · Δ decided by: reasoning
Count the cost difference - that’s the whole pitch, and the whole open question.
Left: 0 tiny gradient steps so far (learning rate η = 0.35). Right: one reasoned jump, 0 forward passes before the edit.
L2 view: Today’s training nudges weights millions of times. A weight-space edit picks the destination in one go - the cost is thinking, not stepping.
TEACHING SCHEMATIC - the “hill” is invented. The step/jump contrast mirrors the paper’s method comparison, but no real loss curve looks this tidy.
Gradient descent nudges every weight a tiny step downhill on the loss, repeated millions of times.
- •
- η (eta) - learning rate, the step size
- •
- ∇L - the gradient: which direction reduces the loss
- •
- t - training step number
Example: PPO needed 100M–500M steps to reach 15.8–25.3 on MinAtar Breakout.
LaTeX code
\theta_{t+1}=\theta_t-\eta\,\nabla_\theta\mathcal{L}(\theta_t)An LLM reads the weights, decides a full edit Δ in one reasoned shot, and writes θ′ - no gradient descent.
- •
- θ′ - the edited weights
- •
- Δ (delta) - the edit, produced by reasoning
- •
- spec - the task description
- •
- feedback - optional forward passes the editor may run
Example: Exp 4: one Δ took a broken ramp-climber from 0/256 to 1,024/1,024 successes.
LaTeX code
\theta'=\theta+\Delta,\quad \Delta=\mathrm{LLM}\!\left(\theta,\ \text{spec},\ \text{optional feedback}\right)① Optimization (SGD, Adam, Muon). The default. Compute how wrong the model is, work out which direction each parameter should move to be slightly less wrong, take a tiny step, repeat - typically millions of steps over billions of examples. It is the only method that has ever produced frontier models; it is also slow, data-hungry, and gives you no human-readable explanation of what changed.
② Trained writers. Hypernetworks (Ha et al., 2016) are networks that output other networks’ weights. Fast-weight programmers (Schlag et al., 2021) let a transformer write temporary weights as it runs. Neural network diffusion (Wang et al., 2024) generates whole weight sets from a learned distribution. All share one requirement: someone had to train the writer first, on lots of examples of networks.
③ Compilers. Tracr (Lindell et al., 2023) compiles a program written in RASP directly into transformer weights - a genuine “weights from recipe” path, but only for the formal, hand-written programs and toy architectures it supports.
④ Weight-space programming (WSP). The paper’s claim: a general-purpose LLM, given the weights (or slices of them) plus a plain-language task, produces an edit Δ by reasoning - optionally checking itself with a handful of forward passes. The writer was never trained to write weights; it learned to understand code, math, and systems, and transfers that understanding to weight space.
Foundational Refreshers (1)