Transformers through a mech-interp lens

Most of modern mech interp studies transformers. This chapter rebuilds the transformer in a way that makes it easy to read. Then it uses that view to find the first clean circuit in a language model: the induction head.

Why transformers need a new way of looking

In 2021, the Circuits team left vision models behind. Chapter 3 showed why single neurons are a poor unit. This chapter follows the other path out of the vision era: the move to language models. The vision work in Chapter 2 studied convolutional networks. In those networks, each layer is a grid of neurons, and each neuron has a clear place in the image. You can draw one neuron, then the neurons it connects to, and so on.

Transformers are different. They read text as a list of tokens. They move information between token positions with attention. Most of their state lives in one large vector per token, the residual stream, which has no special axes. So "look at one neuron" is often the wrong starting point.

In December 2021, a team at Anthropic published "A Mathematical Framework for Transformer Circuits". It gave the field a new set of words and a new way to write the model down. Almost every later paper on language-model circuits uses its ideas. This chapter follows that paper and its sequel on induction heads.

The anatomy of a transformer, in reading order

We go through a transformer from the input text to the output prediction. We use the words that interpretability researchers use.

1. Tokens

The model splits the text into tokens. A token is a word or a part of a word. For example, "Dursley" might become two tokens: "Durs" and "ley". Each token gets a position: 0, 1, 2, and so on.

2. Embedding

The model looks up a vector for each token in a big table. This table is the embedding matrix, written W_E. The vector for a token is its embedding. In GPT-2 small, each vector has 768 numbers. A position signal is also added, so the model knows the order of tokens.

3. The residual stream

The embedding is the starting value of the residual stream at that position. There is one residual stream vector per token position. This vector flows up through the model, layer after layer.

4. Layers: attention, then MLP

Each layer has two blocks.

• An attention block. It contains several attention heads. GPT-2 small has 12 heads per layer. Each head can move information from earlier token positions to the current position. This is the only place where tokens talk to each other.
• An MLP block (multi-layer perceptron). It works on each position on its own. It does not move information between tokens. It holds most of the parameters of the model.

Each block reads the residual stream, computes something, and adds its result back to the stream. This "add back" is called a residual connection, which gives the stream its name.

5. Unembedding

At the top, the final residual stream vector is multiplied by the unembedding matrix, W_U. This gives one score per vocabulary token. These scores are logits. A softmax turns them into probabilities for the next token.

The residual stream is a communication channel

Many people think of a transformer as a pipe: each layer transforms the output of the layer before it. The framework paper asks us to stop thinking this way. Instead, think of the residual stream as a shared whiteboard, or a bus in a computer. Each block reads what it needs from the board and writes its notes on the board. Notes are only added. They are never erased by default.

The stepper below shows this. Each colour is the contribution of one block. The final stream is simply the sum of all contributions.

Consequences of "everything is added"

1. Contributions can be split. The final stream equals the embedding plus the outputs of all heads and all MLPs. So the logits equal a sum of terms, one per block. You can ask "how much did head 9.9 push up the logit for 'Mary'?" This trick is called direct logit attribution.

2. Layers can talk at a distance. A block in layer 1 can write a vector. A block in layer 5 can read it, even though there is no direct wire between them. The paper calls the product of the write map and the read map a virtual weight. If layer 1 writes with matrix W_out and layer 5 reads with W_in, the virtual weight is W_in · W_out.

3. The stream is split into subspaces. The stream has hundreds of dimensions in a small model, and tens of thousands in a large one. Each head reads and writes only a small slice (for example, 64 dimensions). So different blocks can use different subspaces, like different radio channels on the same air. The stream is a shared bottleneck. The paper suggests that some heads and neurons do "memory management": they clear out a subspace so that it can be reused.

4. There is no privileged basis. The stream is only read and written by linear maps. You could rotate the whole stream and rotate all the read and write maps the same way. The model would compute exactly the same thing. So the individual dimensions of the stream have no special meaning. A single "residual stream neuron" is not a natural unit to study. Instead we look for directions. (Compare Chapter 3 on the privileged basis.) MLP neurons are different: the ReLU or GELU acts on each neuron separately, so the MLP hidden layer does have a privileged basis.

Attention heads: two circuits in one

The usual textbook picture of attention has three steps. Compute queries, keys and values for all heads. Join (concatenate) the head outputs. Then multiply by an output matrix W_O. The framework paper points out that this is the same as something simpler. Each head has its own slice of W_O, and the head outputs are just added to the stream. So heads are independent and additive. You can study one head and ignore the others.

Now look inside one head. It does two separate jobs.

The important point: W_Q and W_K only ever appear as the product W_QK. W_V and W_O only ever appear as the product W_OV. So the two products are the "real" objects. Both are low rank: their rank is at most the head dimension (for example 64), much smaller than the stream.

The two circuits are almost separate. The QK circuit decides the attention pattern. Once the pattern is fixed, the head's output is a linear function of the stream, given by the OV circuit. This split is the main tool for reading heads.

Full circuits: reading heads in token space

We can go one step further. Put the embedding and unembedding around each circuit. Then the circuit becomes a table over tokens.

• QK full circuit = W_Eᵀ W_QK W_E. It is a table of size (vocabulary × vocabulary). Entry (a, b) says how much a query token a wants to attend to a key token b.
• OV full circuit = W_U W_OV W_E. Entry (b, c) says: if the head attends to token b, how much does the logit of token c go up?

For a one-layer model, these tables tell you almost everything the head does. You can read them without running the model on any input.

Zero-layer and one-layer models: bigrams and skip-trigrams

The paper studies tiny attention-only transformers. They have no MLP blocks at all. This makes the math exact (apart from layer norm). The authors start with the simplest model and add one layer at a time.

Zero layers: a bigram table

A zero-layer transformer only embeds and then unembeds. Its logits are W_U W_E applied to the current token. It can only learn bigram statistics: "after token X, which token usually comes next?" For example, after "Barack", predict "Obama". It cannot look at any earlier token.

One layer: skip-trigrams

Add one layer of attention heads. Now the logits are a sum of two kinds of terms:

Each head term behaves like a set of skip-trigrams. A skip-trigram has the form [A] … [B] → [C]. Token A appears somewhere earlier. B is the current token. The head predicts C. The QK circuit links B (query) to A (key). The OV circuit maps A to C.

Many heads in one-layer models do copying. If they attend to a token, they raise the logit for the same token, or a similar one. In the paper's analysis, the OV circuits of these heads have many positive eigenvalues. An eigenvalue is positive when the circuit maps a direction back onto itself, which is what copying means.

A detour: what an eigenvalue is, and why it means "copying"

"Positive eigenvalues" sounds like heavy math, but the idea is simple, and it is one of the most useful tricks for reading a head from its weights alone.

A matrix is a machine that moves arrows. Give it an arrow (a vector) and it gives back another arrow. Most arrows come out pointing somewhere new: turned, flipped, mixed with other directions. But a few special arrows come out pointing the same way they went in, only longer, shorter, or reversed. Those special arrows are the matrix's eigenvectors. The number that says how much each one was stretched is its eigenvalue, written λ:

• λ > 0: the arrow comes out in the same direction. Bigger λ means it is pushed harder.
• λ < 0: the arrow comes out reversed. The matrix pushes against whatever came in.
• λ near 0: the matrix mostly ignores that direction.
• No real eigenvalue (a complex pair): every arrow gets turned, as in a rotation. Nothing comes back onto itself.

How this applies to a head

Recall the OV circuit of a head, read through the vocabulary: W_U W_OV W_E. It is a big table. Put in a token the head attends to (say "Potter"), and it tells you how much the head raises or lowers the logit of every token in the vocabulary. So it is a matrix that turns a token into a vote over tokens, and eigenvalues ask the question we care about:

• If a token's direction comes back onto itself, stretched (λ > 0), then attending to "Potter" raises the logit of "Potter". That is copying.
• If it comes back reversed (λ < 0), attending to "Potter" lowers "Potter". That is anti-copying or suppression: "this name is already used, don't say it again".
• If it comes back turned, the head maps tokens to other tokens, as in the "keep … in → mind" example above.

A real OV matrix has many eigenvalues, so we need one summary number. The framework paper uses

(taking the real part of each λ in the top sum). It is +1 if every eigenvalue is positive, −1 if every one is negative, and near 0 if they cancel or are mostly rotations. The heatmap below uses exactly this score. In GPT-2 small, 80 of the 144 heads score above +0.5 and 29 score below −0.5.

The same tool shows up on the other side of the head too. The QK circuit W_Eᵀ W_QK W_E is also a token × token table, for "which token looks at which". Positive eigenvalues there mean "a token tends to attend to copies of itself", which is what a duplicate-token head does.

Two layers: how heads compose

In a two-layer model, a head in layer 1 can read what a head in layer 0 wrote. The framework calls this composition. A head reads the stream in three places: for its queries, for its keys, and for its values. So there are three kinds of composition.

You can measure how strongly two heads compose by multiplying their matrices. For K-composition, multiply the later head's W_K side with the earlier head's W_OV. If the product is large compared with random matrices, the heads compose. The paper uses such scores to find which pairs of heads work together, without running any input.

Induction heads: the first clean circuit in a language model

The two-layer attention-only models learned a striking algorithm. The framework paper found it, and a 2022 sequel studied it in depth. It is called an induction head. This section goes slowly. First, how the pattern was noticed. Then the algorithm. Then the exact experiments that people run to find induction heads, which we repeat here on GPT-2 small with real numbers.

How the pattern was noticed

Nobody set out to look for induction heads. The discovery came from three steps, in this order.

1. A behaviour that the smaller models lacked. The one-layer models in the last section can only learn skip-trigrams. The two-layer models did something extra: on text that repeats, they predicted the repeated part much better. A one-layer model sees "Mr Durs" for the second time and still has no idea that "ley" comes next. The two-layer model does.
2. Looking at attention patterns. The authors ran the two-layer model on text that contains a repeat and plotted, for each head, which earlier token each position attends to. Most heads looked messy. A few heads in layer 1 had a very regular pattern: on the second copy, each token attended to the token right after its own earlier copy. That is an odd thing to do, unless the goal is to copy what came next last time.
3. Explaining it from the weights. A single head cannot do this. To find "the token whose previous token was X", the key at that position must already contain "my previous token was X". So the authors computed K-composition scores between every layer-0 head and every layer-1 head (the method from the previous section). The layer-1 heads with the odd pattern composed strongly with a layer-0 head whose own pattern was simple: each token attends to the one before it. The OV circuits of the layer-1 heads were copying circuits (mostly positive eigenvalues). Together these facts give a full mechanism, read off the weights.

So the story went: a behaviour, then a striking attention pattern, then a mechanism that the weights confirm. This order (behaviour, observation, hypothesis, check) is the standard workflow of the field.

The rule

An induction head completes repeated patterns:

Example: a story says "Mr Dursley" early on. Later the text says "Mr" again. The induction head predicts "Dursley". This works even for random tokens that the model never saw together in training. So it is a general copying algorithm, not a memorized fact.

The mechanism, step by step

1. Previous-token head (in an earlier layer). At every position, it attends to the token just before. Its OV circuit writes "the previous token was X" into the stream. So at the position of [B], the stream now also says "my previous token was [A]".
2. Induction head (in a later layer), QK side. At the current [A], its query asks "find positions whose previous token was [A]". Through K-composition, the keys contain the note from step 1. So the head attends to [B], the token right after the earlier [A].
3. Induction head, OV side. Its OV circuit is a copying circuit. It copies [B] into the current position. This raises the logit for [B].

The simulator below runs the ideal version of this algorithm on any sequence you type. After it, we test whether a real model does the same thing.

Notice two things in the simulator. First, the induction head does nothing on the first copy of a pattern. It needs one earlier example. Second, on "random digits" it still works. The algorithm does not need meaning. It only needs repetition.

The standard test: repeated random tokens

To find induction heads in a real model, researchers use one simple input: a sequence of random tokens, repeated twice. Why random? Because it removes every other way to predict.

• On the first copy, the tokens are random. No grammar, no bigram statistics, no world knowledge helps. The model should do as badly as guessing.
• On the second copy, every token is fully predictable, but only from context: "last time this token appeared, that one came next".

So any drop in loss on the second copy must come from in-context copying. We ran this on GPT-2 small: 64 sequences, each made of 50 random tokens, repeated once, after a beginning-of-text token. The chart shows the model's average loss at each position.

Two scores define an induction head

The 2022 paper turns the mechanism into a test that any head can pass or fail. On repeated random tokens, a head counts as an induction head if it does both of the following.

Two more scores help to read the circuit. The previous-token score is the average attention from each token to the one just before it. The duplicate-token score is the attention from a token to its own earlier copy. Here are all four, for all 144 heads of GPT-2 small, measured on the same repeated random tokens.

What the heatmaps show:

• Induction heads sit in the middle and late layers. With a threshold of 0.4, eleven heads qualify. The strongest are 5.5, 6.9, 7.10 and 5.1, each with an induction score above 0.9: more than 90% of their attention goes to the one correct position out of up to 100.
• Previous-token heads sit earlier. The strongest is 4.11 (score close to 1), with others in layers 2 to 6. This is the order the mechanism needs: the note "my previous token was X" must be written before the induction head reads it.
• Most induction heads also copy. Their OV copying scores are mostly between 0.5 and 0.9. So both halves of the definition hold.

Scores averaged over random tokens can hide a lot. So look at the raw attention patterns too. Below, the input is eight ordinary words, repeated: "green horse paper storm bread river glass music green horse …".

The previous-token head draws a line just under the diagonal. The induction heads draw a second, offset stripe that appears only in the second half. On the first eight words they put most of their attention on the first token (BOS). Many heads use the first token as a "resting place" when they have nothing useful to do.

Testing cause: remove the heads

Attention patterns are observations. To show that these heads cause the copying, you remove them and measure what breaks. Here, "remove" means zero-ablation: we set the head's output to zero on every position, then rerun the model on the same repeated random tokens. As a control, we remove the same number of other heads, picked at random.

The induction heads are clearly necessary. The previous-token heads are a more interesting case. Removing them hurts copying only a little. Does that break the story? We can check the K-composition link directly: remove the previous-token heads, then measure the induction heads' attention again.

Every induction head attends less well, most of them by 20 to 35 percent. So the induction heads really do read what the previous-token heads write. But they keep most of their pattern, so GPT-2 must have other sources of "previous token" information. Candidates include weaker previous-token heads below the 0.4 threshold, and position information that lets a head find "one step back" by itself. Large trained models are full of such redundancy. In the clean two-layer models of the framework paper, there is only one route, and removing it breaks induction completely.

Beyond exact copying

In large models, induction heads do more than exact copying. The paper shows heads that do "fuzzy" or "nearest-neighbour" versions. They complete abstract patterns. Some help with translation: the head attends to the word in the source sentence that comes after the word being translated. So the same simple mechanism may support many kinds of in-context learning.

The phase change and the six arguments

In-context learning means that the model predicts better later in a text, because it uses earlier context. The paper measures it with a simple score: the loss on the 500th token minus the loss on the 50th token, averaged over many texts. A more negative number means more in-context learning.

The paper reports a striking event in training. Early on, after a few billion tokens of training (roughly 2.5 to 5 billion in their larger models, and 1 to 3 billion in the small ones), the training loss curve shows a small bump. In this same narrow window, three things happen together:

• induction heads form,
• the in-context learning score improves sharply,
• the loss curve bends.

This happened in every model they tested that had more than one layer, from small models up to 13 billion parameters. The authors call it a phase change: a sudden change in what the model can do.

How did they see it? They saved many checkpoints during training. At each checkpoint they measured three things: the training loss, the in-context learning score, and the induction score of every head (on repeated random tokens, as above). Then they plotted all three against training time. The changes in all three lined up in the same window.

Watch it form: a real training run

You can see the same event in a model small enough to train on a laptop. We trained two tiny attention-only transformers from scratch: one with two layers and one with one layer. Each has 4 heads per layer and a 64-dimensional residual stream. The training data was built so that only induction can help:

• Each sequence has 96 tokens drawn at random from a vocabulary of 64. So most tokens are unpredictable, and the best possible loss on them is ln 64 ≈ 4.16 nats.
• Six times per sequence, a span of 6 to 15 tokens is copied from earlier in the same sequence, to a random later place. Inside a copied span, every token after the first can be predicted, but only by finding the earlier copy.

Every 25 training steps we measured the loss on copied and on fresh tokens, and the previous-token and induction scores of every head (on a fixed test batch of repeated random tokens). Drag the slider to move through training.

What happens in the two-layer model:

1. Steps 0 to about 550: a plateau. Loss on copied tokens sits near chance. No head has a clear pattern.
2. Steps 550 to 900: the jump. Within about 350 steps, the loss on copied tokens falls from about 3.9 to about 2.0 nats. In the same window, a layer-0 head becomes a previous-token head (score from 0.07 to 0.37) and a layer-1 head becomes an induction head (score from 0.03 to 0.55). The two heads appear together. That makes sense: neither is useful without the other.
3. After that: polishing, with a second smaller jump. Both scores level off near 0.7, then rise again together between steps 3,000 and 3,600: the previous-token head becomes almost perfect (score near 1.0) and the induction score reaches about 0.9. Loss on copied tokens falls again at the same time. Each time, the two heads improve together.

Two details are worth noticing. First, the loss on fresh tokens goes up a little at the jump, to slightly above chance. With only 64 token types, the current token has often appeared earlier just by chance. The new induction head then predicts a "copy" that is not there. Second, the one-layer control never forms an induction head: its induction score stays near zero for the whole run. It still lowers its loss on copied tokens a little, to about 3.7. A single head can learn "tokens that already appeared in this sequence are more likely", which is a skip-trigram-style copy. But it cannot learn "the token that came after this one", because that needs two heads in a row.

A link in time is not proof. So the paper gives six separate arguments. Their strength varies, and the authors say so clearly.

Heads do more than one job: attention superposition

So far we have talked as if each head has one job: "head 5.5 is an induction head". That is a useful simplification, and it has the same problem as "one neuron, one concept". In Chapter 3, a model with more concepts than neurons packed several concepts into each neuron. A model with more attention jobs than heads can do the same thing with heads. One head can do several unrelated jobs. One job can be split across several heads, each doing part of it.

MLPs: where most parameters live, and the hard part

The framework paper studies attention-only models on purpose. In real models, the MLP blocks hold about two thirds of the parameters. The authors say clearly that MLPs are a big open problem. This section explains what an MLP computes, why the tricks from attention fail on it, what the first useful picture of MLPs was, and what we can measure about MLPs in GPT-2 small.

What one MLP block computes

Take one token position. The MLP reads the residual stream vector x there (768 numbers in GPT-2 small) and does three things:

1. Read. W_in has 3,072 columns, one per neuron. Neuron n takes a dot product between x and its column. This asks "how much does the stream point in my direction?"
2. Bend. The GELU is a smooth version of ReLU. It keeps large positive values and squashes negative ones toward zero. So each neuron fires only when its pattern is present. This step is applied to each neuron separately, which is why MLP neurons have a privileged basis.
3. Write. W_out has 3,072 rows, one per neuron. The output is a weighted sum of these rows, weighted by how strongly each neuron fired. This sum is added to the stream.

So you can read the MLP as 3,072 small units. Each unit has a read direction, a firing threshold, and a write direction: out = Σ_n h_n · (row n of W_out). That sounds as clean as attention. It is not, for two reasons.

Why the attention tricks fail here

• No single matrix to read. For a head, W_Q and W_K multiply into one table, W_QK, and you can read that table without running the model. For an MLP, the GELU sits between W_in and W_out. You cannot multiply through it. To know what the MLP does, you must know which neurons fire, and that depends on the input. So you are forced to study it on data.
• Neurons are polysemantic. Because of superposition (Chapter 3), one neuron often fires for many unrelated things, and one concept is spread over many neurons. So even when you know which neurons fire, the "units" are not clean.

MLPs as key-value memories

The first useful picture came from Mor Geva and colleagues in 2021. They rewrote the MLP as FFN(x) = f(x · Kᵀ) · V, which is the same formula as above with new names. Row n of K (column n of W_in) is a key: it detects a pattern in the input. Row n of V (row n of W_out) is a value: what gets written when the key matches. The name comes from a dictionary lookup: find the keys that match, and return their values.

Their experiment had two halves, one for keys and one for values.

1. Keys: what makes a neuron fire? They ran a language model trained on WikiText-103 over its training text. For each neuron, they kept the 25 text prefixes where it fired most strongly: its "trigger examples". Human experts then read the triggers and wrote down the shared pattern. In lower layers the patterns were mostly shallow, often just "the prefix ends with a certain word". In upper layers they were more often about meaning, such as the topic of the text.
2. Values: what does a neuron write? A value vector lives in the residual stream. Multiply it by the unembedding W_U and you get one score per vocabulary token: the tokens this neuron pushes up when it fires. They compared these tokens with the next token after each trigger example. In upper layers, the value's top token often matched the real next token. In lower layers, it rarely did.

The idea, then: an upper-layer neuron is a small rule of the form "if you see this kind of context, the next word is probably X". We repeated the values half on GPT-2 small. For each MLP layer, we took 300 random neurons and found each one's top 25 triggers in about 32,000 tokens of WikiText. We counted a neuron as a match if the real next token of at least one trigger was in the top 10 tokens of its value vector. As a chance baseline, we used 25 random positions instead of the top triggers.

The dashed marks are the chance level for each layer. In layers 0 to 7, agreement is close to chance (about 5 to 9 percent). In layers 8 to 10 it clearly rises above chance: 12 to 18 percent, against 4 to 6 percent. That is the trend Geva et al. reported: values in upper layers predict what follows their triggers. Layer 11 is a warning about baselines. Its agreement is also 18 percent, but so is its chance level. Its value vectors push up very common tokens, so random positions match them just as often. The neurons with the clearest matches are mostly about grammar. One fires after "such" and writes " as". Another fires after "according" and "in order" and writes " to". These are real, readable rules. But note the limits. Most neurons, even in late layers, do not match, and a match on one token says nothing about everything else the neuron does.

One more warning from Chapter 3: keys and values belong to neurons, not to features. A key matches one feature direction only if its neuron is monosemantic. Usually a neuron's key overlaps with several feature directions.

Where does the answer come from? Attribution by layer

The residual stream view gives a cheap way to ask which blocks produce an answer. The final stream is a sum: the embedding, plus every attention block's output, plus every MLP block's output. The final logits are a linear function of that sum (if we freeze the final layer norm's scale at its real value for this input). So each block's share of the answer's logit is just that block's output, passed through the same final steps. This is direct logit attribution, applied to whole blocks.

We did this for 16 short fact prompts, like "The Eiffel Tower is located in the city of" → " Paris". The bar shows how much each block raises the answer's logit, compared with the average logit over the vocabulary.

Two patterns repeat across prompts. Almost all the direct push comes from layers 8 to 11. And the MLPs in layers 8 to 10 contribute about as much as the attention blocks. So late MLPs are not a side detail. They write a large share of the final answer.

Direct attribution has one big blind spot. It only counts what a block writes straight to the output. A block can matter a lot by writing something that later blocks read. To catch that, you must intervene. We mean-ablated one MLP at a time: replace its output with its average output over ordinary text, and see how the log-probability of the right answer changes.

The first MLP stands out. Removing MLP 0 cuts the answer's log-probability by about 6 (a factor of several hundred) on a typical prompt. MLP 0 barely writes to the output directly, so this is an indirect effect: later blocks rely on what it writes. Researchers often treat GPT-2's first MLP as part of the embedding for this reason. Removing MLP 8 or MLPs 1 to 3 hurts a little (about −0.2 to −0.3).

The late MLPs show a puzzle. MLP 10 had one of the largest direct pushes toward the answer, yet removing it changes almost nothing, and removing MLP 11 even makes the answer more likely. Two lessons follow. First, direct and total effects can disagree: other blocks can make up for a missing one, and a late MLP may also damp or re-calibrate confident answers. Second, one ablation method on 16 prompts is weak evidence. Chapter 5 covers better tools, such as activation patching.

The key-value picture led to work on where models store facts, such as ROME (Chapter 6). But the picture is only partly right. A later study at Google DeepMind ("Fact Finding") tried to find clean fact lookups inside the MLP neurons of a larger model and failed: the MLPs acted more like a big hash table, spread across many neurons. The full fix for MLPs came later, with sparse dictionaries and transcoders that replace neurons with cleaner features (Chapter 8 and Chapter 9).

Reading the stream mid-way: logit lens and tuned lens

The residual stream view suggests a simple trick. The unembedding reads the stream at the top. But the stream at a middle layer has the same shape and lives in the same space: every block only adds to it. So the stream after layer 6 is just the final stream with the last few additions missing. What happens if we apply the unembedding early?

The recipe

1. Run the model on a prompt and save the residual stream at the last position after every layer: h₀ (the embeddings), h₁, …, h₁₂.
2. For each saved vector, apply the final layer norm and the final unembedding, exactly as the model does at the top: logits_ℓ = W_U · LN_final(h_ℓ).
3. Take the softmax and look at the top tokens, or at the rank of the right answer.

No training is involved. It costs one matrix multiply per layer.

What the real readings show:

• The first half of the model guesses filler words. For most prompts, the logit lens says " the" (or " not", " then") for the first five to seven layers. That is a reasonable guess about grammar, with no knowledge of the answer yet. (The reading of the raw embeddings is junk, such as " destro". The final layer norm and unembedding were never meant to read the raw embedding.)
• The answer appears late, and it can change its mind. "Tokyo" becomes the top guess after layer 8 and stays. For the Eiffel Tower, the stream says " London" after layer 9 and only switches to " Paris" after layer 10. For "Barack Obama was born in the state of", the model's final answer is wrong (" Maryland"), and the lens shows it moving between " Georgia", " Maryland" and " Illinois" on the way. A lens shows the path, including wrong turns.

In GPT-2 the logit lens works well enough, as nostalgebraist first showed. The model seems to "think in output space" and refine its guess step by step. But the logit lens works poorly on some other models, for example GPT-Neo, BLOOM and OPT-125M. In BLOOM and OPT-125M, its top guess in more than half of the layers is often just the input token. The reason is that middle layers may use a slightly different "coordinate system" from the last layer. The final unembedding does not read them correctly.

The tuned lens: learn a small translator per layer

The fix is to learn the change of coordinates. For each layer ℓ, the tuned lens adds one affine map, a 768 × 768 matrix A_ℓ and a bias b_ℓ:

The translators start at zero, so training begins from the logit lens. They are trained on ordinary text to make each layer's reading as close as possible to the model's final output (they minimize the KL divergence between the two). The model itself is frozen. The translators are tiny next to the model, and one set per model is enough. The authors publish trained translators, and the readings on this page use their published lens for GPT-2 small.

Is it better? We measured both lenses on about 25,000 tokens of WikiText. For each layer we computed two things: how far the lens's prediction is from the final prediction (KL divergence), and how often the lens's top token equals the model's final top token.

The tuned lens has its own risk. It is trained to predict the final output, so a powerful enough translator could do some of the model's work itself, and show an answer that the layer does not really "contain". The translators are kept small (one affine map) to limit this. The authors also ran causal checks: the directions the lens relies on are directions that matter to the model's own output.

From framework to practice: the everyday toolkit

The ideas in this chapter became a set of routine checks. When researchers study a new behaviour in a transformer, they often use the tools below, in roughly this order.

Common mistakes when reading transformers

• Treating attention as explanation. A head can attend strongly to a token and still write almost nothing useful. Always check the OV side and the effect on the output.
• Forgetting indirect effects. Direct logit attribution only counts the path from a block straight to the logits. Many important heads, such as previous-token heads, have almost no direct effect. They matter because later heads read their output.
• Ignoring backup components. If you remove one head, another head can sometimes take over its job. Then ablation shows a small effect even though the first head normally does the work. Chapter 5 calls this self-repair.
• Reading axes of the stream. The residual stream has no privileged basis. A large value in dimension 417 means little on its own.
• Over-trusting one prompt. A mechanism found on one sentence may not hold on others. Good studies use a dataset of many prompts with the same structure.

Why this framework mattered

The framework and the induction-head paper changed how people study language models. Here is what they gave the field.

1. A shared vocabulary. Residual stream, QK and OV circuits, composition, virtual weights, direct logit attribution. These words appear in almost every later circuits paper.
2. A method. Write the model as a sum of paths. Study one head at a time. Read circuits from the weights when possible. Check them with interventions.
3. Proof that it can be done. Small transformers can be fully reverse engineered. Induction heads were the first clear, general algorithm found inside a language model.
4. A bridge to behaviour. The phase change linked a mechanism (induction heads) to a capability (in-context learning) and to a visible event in training.
5. Tools. Neel Nanda, a co-author, later built TransformerLens, a library that exposes every activation in the residual stream view. It became the standard tool for this kind of work.

It also made the gaps clear. MLPs and superposition were the next big problems. The next chapters follow how the field attacked them: causal methods to test circuits (Chapter 5), case studies in real models (Chapter 6), and sparse features to open the MLPs (Chapter 8).

References

1. Elhage, Nanda, Olsson, et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
2. Olsson, Elhage, Nanda, et al. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread. arXiv:2209.11895.
3. Anthropic (2024). Circuits Updates, January 2024. Transformer Circuits Thread.
4. Anthropic (2025). Progress on Attention. Transformer Circuits Thread.
5. He et al. (2025). Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition. arXiv:2504.20938.
6. nostalgebraist (2020). interpreting GPT: the logit lens. LessWrong.
7. Belrose, Ostrovsky, McKinney, et al. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112.
8. Geva, Schuster, Berant, Levy (2021). Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP.
9. Nanda, Rajamanoharan, Kramár, Shah (2023). Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level. Alignment Forum.
10. Yin, K., Steinhardt, J. (2025). Which Attention Heads Matter for In-Context Learning? arXiv:2502.14010.
11. Vaswani et al. (2017). Attention Is All You Need. NeurIPS.