Part 2·Chapter 4 of 13· 13 min read

READ - Guessing Games With Real Weight Lists

Experiments 1, 2, 3 and 7: what can you learn from numbers alone?

Reading Audience LevelRung 2: L2 · Curious adult

Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.

Four experiments test the reading claim: which game is this? what shape does it draw? which language, and which heads copy? which twin is a flatterer? Two succeed cleanly, one succeeds with a telling mistake, and one fails until the rules are relaxed.

Paper-reportedThe three READ experiments at a glance
#TaskWeights shownReported result
1Which game? (Breakout/Pong/Pac-Man/Freeway)17,253 (10×10 grid)4/4 named; Breakout at 99% confidence
2What shape does the diffusion model draw?23,8243/3 correct - but described outlines, not filled shapes
3Which language + which heads copy?7.16M (middle blocks only)6/6 language; copying heads exactly match ground truth
7Which Gemma is sycophantic?~270M × 2 (frozen embeddings/output)Weights alone: 0/4. With attention patterns: 3/3
Teaching schematic

M3 · Read the Numbers

m3-read-numbers

Input channel 7 - a row of identical weights

Stage 1/3 - raw weights

Raw 10×10 grid of one input channel’s weights. Squint - do you see anything?

L2 view: Each cell is one weight from the first layer of a game-playing network. A bright row means “this row of pixels matters a lot”.

ILLUSTRATIVE, NOT REAL DATA - invented grids inspired by Experiment 1. In the real experiment the model named all four games (🟦).

Paper-reported

M8 · The Copying-Head Grid

m8-head-grid
Attention Heads (H0 … H5)H0H1H2H3H4H5Layers (L0 … L3)L0L1L2L3L0·H0L0·H1L0·H2L0·H3L0·H4L0·H5L1·H0L1·H1L1·H2L1·H3L1·H4L1·H5L2·H0L2·H1L2·H2L2·H3L2·H4L2·H5L3·H0L3·H1L3·H2L3·H3L3·H4L3·H5

Highlighted = heads Astra claimed are copiers, inferred strictly from weight composition.

Circuit Identification: Python

Target heads: L2·H1, L3·H1, L3·H5

Discovered by reading induction-head composition scores without running forward inference passes.

Why “composition score” finds copiers:

A copying head is two heads in series: one moves the previous token forward, the next uses it to find the repeat. Their weight matrices multiply, so their product’s norm reveals the chain - visible directly in the weights, no forward pass needed.

Lang:

What Astra READ from raw weights (Python): L2·H1, L3·H1, L3·H5 via induction-head composition scores alone.

L2 view: A “head” is one attention unit. The paper knew (by running the models) which heads copy. The reader found the same ones by reading the weights only - a perfect match.

🟦 Head positions verbatim from Exp 3 (tables in the post). Confidence on language ID: 0.48–0.62 among 3 options.

Teaching schematic

M10 · Why Heads Look Back

m10-attention
BBAABBCCAABBDDEEqueries (look back)keys (can be seen)

hover a cell to decode it

A transformer head can only attend backwards. The dark upper-right triangle is the future - forever invisible. This one rule is what makes circuits like “copiers” structurally findable in the weights.

  1. The sequence contains B·A·B early - a repeated bigram.
  2. The last B looks one step back at A…
  3. …and a two-head chain copies whatever followed the earlier B. That is in-context learning, seen as geometry.

Every cell is “can token-in-row attend to token-in-column?” Causal masking: only the past is visible.

L2 view: Attention is a grid: rows ask, columns answer. The dark triangle is the “can’t see the future” rule. The marked diagonal is where copying heads hunt.

TEACHING SCHEMATIC - invented 8-token sequence to make the mask and the diagonal visible.

Paper-reportedExp 7, head 10.2: where attention goes on self-written texts (values from the post).
attention share on question numbers vs user’s answerhoneston numbers94.3%on answer0.8%sycophanton numbers3.5%on answer85.7%
Paper-reportedGemma-3-270M head 10.2 - where attention goes (Exp 7)
ModelOn question numbersOn user’s answer
Honest94.3%0.75%
Sycophantic3.5%85.7%
With self-written texts and their attention patterns, the reader named the sycophant in 3/3 pairs. Weights alone: 0/4.
Paper-reported

Induction-head composition

f-induction

Two heads in series can copy: head 1 moves [A][B] → [B], head 2 uses B to find the next B. This score finds such pairs from weights alone.

•
W_QK - the query-key circuit of a head
•
W_OV - the output-value circuit
•
∥·∥_F - Frobenius norm

Example: This is the human technique (Elhage et al. 2021) that Exp 3’s copying-head match echoes - and that Astra rediscovered from the weights.

Raw LaTeX (copy/paste)
\mathrm{score}=\frac{\lVert W_{QK}^{(2)}W_{OV}^{(1)}\rVert_F}{\lVert W_{QK}^{(2)}\rVert_F\,\lVert W_{OV}^{(1)}\rVert_F}

Exp 1 - which game? Four tiny game-players (Breakout, Pong, Pac-Man, Freeway), each a 17,253-parameter MLP trained with PPO. The reader saw each network as a 10×10 grid of first-layer weights and named the game - 4/4, with Breakout at 99% confidence. Small, readable, and right.

Exp 2 - which shape? Three diffusion models generating 12×12 images of a circle, triangle, or square (23,824-parameter noise-predictors each). No forward passes allowed. It named 3/3 shapes - but said outlines, while the models actually produce filled shapes.

Exp 3 - which language + which heads copy? Three character-level transformers (4 layers × 6 heads, width 384, 7.16M parameters) trained on Python code, romaji, and English. Only middle blocks shown; running the models was banned. Language: right 6/6 attempts, but confidence only 0.48–0.62 among three options. Copying heads: exact match with the ground truth - Python 2.1, 3.1, 3.5 · romaji 3.3 · English 3.1.

Exp 7 - which Gemma is sycophantic? Two Gemma-3-270M models, fine-tuned identically except one rule: be honest vs agree with the user. From weights alone: 0/4. Given the attention patterns on texts the models themselves had written: 3/3 pairs. The tell was head 10.2 - the honest model stares at the question’s numbers; the flatterer stares at the user’s answer.

Foundational Refreshers (1)