Small Models, Big Models - Who Benefits?
Demonstrated, suggested, speculative - by model size.
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
Where could weight-space programming genuinely help? The honest split is by scale: small networks are writable (demonstrated), big models are so far only editable at diagnosed spots (demonstrated once), and everything beyond that is suggestion.
| Side | Use | Status | Why |
|---|---|---|---|
| Small models | Write/initialise tiny specialist nets (game players, controllers) | Demonstrated | Exp 5–6 wrote working policies from scratch |
| Small models | Fix broken small policies (robots, edge agents) | Demonstrated | Exp 4: one-shot 0/256 → 1024/1024 |
| Big models | Targeted fixes of known bad behaviours | Demonstrated (one model) | Exp 8 on Qwen3.5-0.8B; generalisation unproven |
| Big models | Audit / interpret without running the model | Suggested | Exp 3’s copying heads + Exp 7’s partial success hint at it |
| Any model | Zero-data unlearning, patch ecosystems | Speculative | Predictions §5.4 - none tested |
Small models (thousands to a few million weights): the paper demonstrated writing them from scratch (MinAtar, Go) and fixing a broken small policy (the ramp car). These are cases where the whole network fits in the reasoner’s attention window and the task rules are stateable. Practical cousins: simple game bots, home robots’ controllers, signal filters.
Big models (hundreds of millions to billions): demonstrated once - the Qwen math fix, which was less “reading the model” and more “a diagnosed surgery on two MLP blocks”, guided by reasoning plus feedback passes. Everything grander (auditing, unlearning, capability steering at scale) is currently suggested, not shown.
The asymmetry that matters: to write a big model you’d have to hold billions of numbers in working context - nobody claims that yet. To edit one, you only need to find the small part where a problem lives - the demonstrated pattern. Expect “WSP for small nets” products long before “WSP for foundation models”, whatever the headlines imply.
Foundational Refreshers (1)