Part 2·Chapter 7 of 13· 9 min read

How Did It Do It? - Two Mechanisms

reason → program → compile, versus reason → find a direction → subtract it.

Reading Audience LevelRung 2: L2 · Curious adult

Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.

When the edits worked, how did the reasoning actually produce numbers? The paper reports two distinct mechanisms: compiling an understood program into weights (paddle predictor, Go liberties/atari), and locating a harmful direction to subtract (the math-loop fix). This chapter walks both, and flags how much is story versus measurement.

Teaching schematicThe two reported mechanisms: compile-a-program vs subtract-a-direction. ILLUSTRATIVE.
reason → program → compile into weightsreason about the gamewrite pseudo-rules (liberties, atari)compile rules into W₁, W₂, …test 0–50 passesreason → find a direction → subtract itread MLPs of layers 15 & 19identify the re-injected “loop” usubtract u from 64 columns × 2verify: token-limit share falls
Teaching schematic

M4 · Compile a Rule into Weights

m4-compile-rule
ballpaddle
action = argmax z = left

Inside the network

h₀ ←
0.35
h₁ →
0.00

Logits

z₀ ←
0.35
z₁ ●
0.05
z₂ →
0.00

W₁ = [[-1, 1], [1, -1]] · b₁ = [-0.05, -0.05] · W₂ = [[1,0],[0,0],[0,1]] · b₂ = [0, .05, 0]

With ball at 0.30 and paddle at 0.70: h₀ (ball left) = 0.35, h₁ (ball right) = 0.00 → logits 0.35 / 0.05 / 0.00 → action: LEFT

L2 view: Reasoning about the game produced a rule; the rule became 17 numbers; the numbers ARE a neural network. No training happened.

TEACHING SCHEMATIC - the paddle net is invented for teaching (it is lab/01). It is NOT one of the paper’s networks.

python
import numpy as np
relu = lambda z: np.maximum(z, 0.0)
W1 = np.array([[-1., 1.], [1., -1.]]); b1 = np.array([-0.05, -0.05])   # h0: ball left of paddle, h1: ball right
W2 = np.array([[1., 0.], [0., 0.], [0., 1.]]); b2 = np.array([0., 0.05, 0.])
ACTIONS = ["left", "stay", "right"]
def act(ball, paddle):
    h = relu(W1 @ np.array([ball, paddle]) + b1)
    return ACTIONS[int(np.argmax(W2 @ h + b2))]
assert act(0.2, 0.7) == "left" and act(0.8, 0.3) == "right" and act(0.5, 0.52) == "stay"
lab/01_compile_a_rule.py - the hand-written miniature of the compile path (also in /lab).
python
import numpy as np
rng = np.random.default_rng(0)
d_model, d_ff = 16, 64
W = rng.normal(size=(d_model, d_ff))
u = rng.normal(size=d_model); u /= np.linalg.norm(u)
W_full = W - np.outer(u, u @ W)                       # (I - u u^T) W
print("max |u^T W'| =", np.abs(u @ W_full).max())      # ~ 1e-16
k = 8; cols = np.argsort(-np.abs(u @ W))[:k]            # sparse variant: edit only k columns
W_sp = W.copy(); W_sp[:, cols] -= np.outer(u, u @ W[:, cols])
print(f"edited {k}/{d_ff} columns; remaining u-energy {np.linalg.norm(u @ W_sp) / np.linalg.norm(u @ W):.2f}")
lab/02_direction_surgery.py - the subtract-a-direction path in 10 lines (also in /lab).

The compile path (WRITE experiments). For the paddle predictor and the Go policy, the reported reasoning runs: understand the game → state the rules you’d follow → compile those rules into weights. In Go the paper says the resulting layers carry liberties and atari concepts - i.e. the LLM’s stated strategy is visible in the weights’ structure. The miniature in this chapter (the paddle net above) is a human’s version of the same manoeuvre, so you can run and inspect it in seconds.

The direction path (the math fix). The reasoning located a direction u that layers 15 and 19 kept re-injecting - a thought loop - then subtracted u from the 64 most loop-carrying output columns in each of those layers. After the subtraction, uᵀW′ ≈ 0 on those columns: the loop’s fuel line is cut. That is exactly the single-direction technique from published refusal-removal work (Arditi et al. 2024) - aimed at a new target.

Foundational Refreshers (1)