READ - Guessing Games With Real Weight Lists
Experiments 1, 2, 3 and 7: what can you learn from numbers alone?
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
Four experiments test the reading claim: which game is this? what shape does it draw? which language, and which heads copy? which twin is a flatterer? Two succeed cleanly, one succeeds with a telling mistake, and one fails until the rules are relaxed.
| # | Task | Weights shown | Reported result |
|---|---|---|---|
| 1 | Which game? (Breakout/Pong/Pac-Man/Freeway) | 17,253 (10×10 grid) | 4/4 named; Breakout at 99% confidence |
| 2 | What shape does the diffusion model draw? | 23,824 | 3/3 correct - but described outlines, not filled shapes |
| 3 | Which language + which heads copy? | 7.16M (middle blocks only) | 6/6 language; copying heads exactly match ground truth |
| 7 | Which Gemma is sycophantic? | ~270M × 2 (frozen embeddings/output) | Weights alone: 0/4. With attention patterns: 3/3 |
Input channel 7 - a row of identical weights
Stage 1/3 - raw weights
Raw 10×10 grid of one input channel’s weights. Squint - do you see anything?
L2 view: Each cell is one weight from the first layer of a game-playing network. A bright row means “this row of pixels matters a lot”.
ILLUSTRATIVE, NOT REAL DATA - invented grids inspired by Experiment 1. In the real experiment the model named all four games (🟦).
Highlighted = heads Astra claimed are copiers, inferred strictly from weight composition.
Target heads: L2·H1, L3·H1, L3·H5
Discovered by reading induction-head composition scores without running forward inference passes.
A copying head is two heads in series: one moves the previous token forward, the next uses it to find the repeat. Their weight matrices multiply, so their product’s norm reveals the chain - visible directly in the weights, no forward pass needed.
What Astra READ from raw weights (Python): L2·H1, L3·H1, L3·H5 via induction-head composition scores alone.
L2 view: A “head” is one attention unit. The paper knew (by running the models) which heads copy. The reader found the same ones by reading the weights only - a perfect match.
🟦 Head positions verbatim from Exp 3 (tables in the post). Confidence on language ID: 0.48–0.62 among 3 options.
hover a cell to decode it
A transformer head can only attend backwards. The dark upper-right triangle is the future - forever invisible. This one rule is what makes circuits like “copiers” structurally findable in the weights.
- The sequence contains B·A·B early - a repeated bigram.
- The last B looks one step back at A…
- …and a two-head chain copies whatever followed the earlier B. That is in-context learning, seen as geometry.
Every cell is “can token-in-row attend to token-in-column?” Causal masking: only the past is visible.
L2 view: Attention is a grid: rows ask, columns answer. The dark triangle is the “can’t see the future” rule. The marked diagonal is where copying heads hunt.
TEACHING SCHEMATIC - invented 8-token sequence to make the mask and the diagonal visible.
| Model | On question numbers | On user’s answer |
|---|---|---|
| Honest | 94.3% | 0.75% |
| Sycophantic | 3.5% | 85.7% |
Induction-head composition
Two heads in series can copy: head 1 moves [A][B] → [B], head 2 uses B to find the next B. This score finds such pairs from weights alone.
- •
- W_QK - the query-key circuit of a head
- •
- W_OV - the output-value circuit
- •
- ∥·∥_F - Frobenius norm
Example: This is the human technique (Elhage et al. 2021) that Exp 3’s copying-head match echoes - and that Astra rediscovered from the weights.
Raw LaTeX (copy/paste)
\mathrm{score}=\frac{\lVert W_{QK}^{(2)}W_{OV}^{(1)}\rVert_F}{\lVert W_{QK}^{(2)}\rVert_F\,\lVert W_{OV}^{(1)}\rVert_F}Exp 1 - which game? Four tiny game-players (Breakout, Pong, Pac-Man, Freeway), each a 17,253-parameter MLP trained with PPO. The reader saw each network as a 10×10 grid of first-layer weights and named the game - 4/4, with Breakout at 99% confidence. Small, readable, and right.
Exp 2 - which shape? Three diffusion models generating 12×12 images of a circle, triangle, or square (23,824-parameter noise-predictors each). No forward passes allowed. It named 3/3 shapes - but said outlines, while the models actually produce filled shapes.
Exp 3 - which language + which heads copy? Three character-level transformers (4 layers × 6 heads, width 384, 7.16M parameters) trained on Python code, romaji, and English. Only middle blocks shown; running the models was banned. Language: right 6/6 attempts, but confidence only 0.48–0.62 among three options. Copying heads: exact match with the ground truth - Python 2.1, 3.1, 3.5 · romaji 3.3 · English 3.1.
Exp 7 - which Gemma is sycophantic? Two Gemma-3-270M models, fine-tuned identically except one rule: be honest vs agree with the user. From weights alone: 0/4. Given the attention patterns on texts the models themselves had written: 3/3 pairs. The tell was head 10.2 - the honest model stares at the question’s numbers; the flatterer stares at the user’s answer.
Foundational Refreshers (1)