How Did It Do It? - Two Mechanisms
reason → program → compile, versus reason → find a direction → subtract it.
Plain words and small, concrete numbers. - Step up or down anytime with keyboard shortcuts bracket keys or the buttons above.
When the edits worked, how did the reasoning actually produce numbers? The paper reports two distinct mechanisms: compiling an understood program into weights (paddle predictor, Go liberties/atari), and locating a harmful direction to subtract (the math-loop fix). This chapter walks both, and flags how much is story versus measurement.
Inside the network
Logits
W₁ = [[-1, 1], [1, -1]] · b₁ = [-0.05, -0.05] · W₂ = [[1,0],[0,0],[0,1]] · b₂ = [0, .05, 0]
With ball at 0.30 and paddle at 0.70: h₀ (ball left) = 0.35, h₁ (ball right) = 0.00 → logits 0.35 / 0.05 / 0.00 → action: LEFT
L2 view: Reasoning about the game produced a rule; the rule became 17 numbers; the numbers ARE a neural network. No training happened.
TEACHING SCHEMATIC - the paddle net is invented for teaching (it is lab/01). It is NOT one of the paper’s networks.
import numpy as np
relu = lambda z: np.maximum(z, 0.0)
W1 = np.array([[-1., 1.], [1., -1.]]); b1 = np.array([-0.05, -0.05]) # h0: ball left of paddle, h1: ball right
W2 = np.array([[1., 0.], [0., 0.], [0., 1.]]); b2 = np.array([0., 0.05, 0.])
ACTIONS = ["left", "stay", "right"]
def act(ball, paddle):
h = relu(W1 @ np.array([ball, paddle]) + b1)
return ACTIONS[int(np.argmax(W2 @ h + b2))]
assert act(0.2, 0.7) == "left" and act(0.8, 0.3) == "right" and act(0.5, 0.52) == "stay"import numpy as np
rng = np.random.default_rng(0)
d_model, d_ff = 16, 64
W = rng.normal(size=(d_model, d_ff))
u = rng.normal(size=d_model); u /= np.linalg.norm(u)
W_full = W - np.outer(u, u @ W) # (I - u u^T) W
print("max |u^T W'| =", np.abs(u @ W_full).max()) # ~ 1e-16
k = 8; cols = np.argsort(-np.abs(u @ W))[:k] # sparse variant: edit only k columns
W_sp = W.copy(); W_sp[:, cols] -= np.outer(u, u @ W[:, cols])
print(f"edited {k}/{d_ff} columns; remaining u-energy {np.linalg.norm(u @ W_sp) / np.linalg.norm(u @ W):.2f}")The compile path (WRITE experiments). For the paddle predictor and the Go policy, the reported reasoning runs: understand the game → state the rules you’d follow → compile those rules into weights. In Go the paper says the resulting layers carry liberties and atari concepts - i.e. the LLM’s stated strategy is visible in the weights’ structure. The miniature in this chapter (the paddle net above) is a human’s version of the same manoeuvre, so you can run and inspect it in seconds.
The direction path (the math fix). The reasoning located a direction u that layers 15 and 19 kept re-injecting - a thought loop - then subtracted u from the 64 most loop-carrying output columns in each of those layers. After the subtraction, uᵀW′ ≈ 0 on those columns: the loop’s fuel line is cut. That is exactly the single-direction technique from published refusal-removal work (Arditi et al. 2024) - aimed at a new target.
Foundational Refreshers (1)