Circuit case studies

Between 2022 and 2024, researchers opened real language models and traced specific behaviours to specific parts. This chapter walks through the most famous of these case studies, one at a time.

How a circuit case study works

Chapter 4 gave the vocabulary (heads, composition, the residual stream). Chapter 5 gave the tools (ablation and patching). This chapter puts them together on real models. All the studies in this chapter follow a similar recipe. If you know the recipe, each study is easier to read.

1. Pick a narrow task. Choose a behaviour that the model does well, and that you can test with many similar prompts. For example: "When Mary and John went to the store, John gave a drink to ___".
2. Pick a metric. Choose one number that measures success. A common choice is the logit difference: the score of the right answer minus the score of a wrong answer.
3. Make a corrupted version. Change the prompt a little so that the right answer is no longer clear. Compare the clean run and the corrupted run.
4. Locate. Use activation patching and ablation to find which components matter. Chapter 5 explains these tools.
5. Interpret. For each important component, ask what it does. Look at where an attention head looks. Project its output onto the vocabulary to see which words it pushes up.
6. Validate. Keep only the circuit, switch off the rest, and check that the task still works. Also check that each part is needed.

The first famous case study used this recipe on a different target: the induction heads (2022). Those heads complete repeated patterns. Chapter 4 explains them, because they are part of the basic toolkit for transformers. Here we start with the next big study.

Notation used in this chapter

We name an attention head by its layer and its index, counting from zero. So 9.9 means "layer 9, head 9". GPT-2 small has 12 layers with 12 heads each, so 144 heads in total. Some papers write the same head as L9H9 or a9.h9.

Indirect object identification: which name comes next?

The task

Read this sentence and guess the next word:

The answer is "Mary". In grammar, Mary is the indirect object (IO): the person who receives the drink. The researchers call this task IOI, for Indirect Object Identification. GPT-2 small does it well.

The sentence has two names. The subject (S, here John) appears twice. We call the two copies S1 and S2. The indirect object (IO, here Mary) appears only once. So a simple rule solves the task:

1. Find the names in the sentence.
2. Find the name that appears twice.
3. Output the other name.

The question was: does GPT-2 really use this rule, and if so, which parts of the model do each step?

How they searched

The metric was the logit difference: logit("Mary") minus logit("John"). A large positive number means the model clearly prefers the right name.

For the corrupted input, they used the ABC dataset. In these sentences, three different names appear, for example "When Mary and John went to the store, Alice gave a drink to". Now no name is repeated. So the information "which name is duplicated" is gone. Any part of the model that carries that information will behave differently on the two inputs.

The authors then used a new method that they called path patching. It swaps in activations from the ABC run along one path at a time, for example only the path from head 7.3 to the queries of head 9.9. If the logit difference drops, that path matters. They also used mean ablation: they replaced a head's output with its average output over the ABC data. This "switches off" the head's task-specific information but keeps its normal size. Chapter 5 explains both methods in detail.

The circuit, step by step

Use the stepper. It builds the circuit one group at a time. The horizontal axis shows token positions. The vertical axis shows layers (0 at the bottom, 11 at the top). Each box is one or more attention heads. A curved line shows where a head looks (attends).

Here is the full list of heads. The heads in brackets are smaller members of their class. Together they make the 26 heads of the paper.

How good is the explanation?

The authors proposed three tests for any circuit. These tests are now standard vocabulary.

• Faithfulness. Does the circuit alone do the task? They mean-ablated every head outside the circuit. The circuit kept 87% of the full model's logit difference.
• Completeness. Does the circuit contain every part that matters? Test: take any subset K of the circuit. Removing K from the circuit should hurt as much as removing K from the full model. Random subsets passed this test. But a greedy search found subsets where the two numbers differed a lot. So the circuit is not fully complete.
• Minimality. Is every part needed? For each head, they found a situation where removing it made a clear difference. So no head was "extra".

Copy suppression: explaining one head completely

The IOI study left a puzzle. Why does head 10.7 push the right answer down? A later study looked at this one head on all kinds of text, not only IOI sentences.

The mechanism

Head 10.7 does copy suppression. It works in three steps:

1. Earlier layers predict some next token, for example a word that already appeared earlier in the text.
2. The head's QK circuit asks: "Is the token that we are about to predict already present earlier in the context?" If yes, the head attends to that earlier copy.
3. The head's OV circuit writes a negative vote for that same token. So the prediction "repeat this word" gets weaker.

Why would a model want this? Earlier heads often copy words from the context. Copying is a good guess, but not always. Copy suppression corrects for naive copying. It makes the model's predictions better calibrated: its confidence better matches how often it is right.

The authors explained 76.9% of head 10.7's effect with this mechanism, using evidence from the weights. This is rare. Most studies explain a head on one task. This study explained one head on the model's whole training distribution.

Copy suppression also explains part of the IOI self-repair. In IOI, the name movers copy "Mary". Head 10.7 sees that "Mary" is predicted and suppresses it a little. If you remove the name movers, "Mary" is no longer predicted strongly, so head 10.7 suppresses less. The logit for Mary partly comes back. Self-repair is partly a side effect of a head that reacts to what other heads do. On one narrow task, the authors found that copy suppression explains 39% of the self-repair.

Greater-than: how GPT-2 compares years

The task

A sensible end year must be later than the start year. So the model should give high probability to 33, 34, …, 99 and low probability to 00, …, 32. The researchers measured this with the probability difference: the total probability on years greater than YY, minus the total probability on years less than or equal to YY. GPT-2 small scores 81.7%. So it really does "know" the comparison.

For the corrupted input, they replaced the start year with "01" (as in "1701"). Then almost every end year is valid, and the comparison information is gone.

The circuit

Path patching found a small circuit:

1. Attention heads in layers 5 to 9 (a5.h1, a5.h5, a6.h9, a7.h10, a8.h8, a8.h11, a9.h1) sit at the final position. They attend to the start-year token "32" and move information about it to the end.
2. MLPs 8, 9, 10 and 11 do the comparison. The researchers looked at their outputs with the logit lens. The MLP outputs push up years greater than YY and push down years less than or equal to YY.
3. MLP 8 acts mostly indirectly: it changes what later MLPs do. MLP 11 pushes up roughly the first 50 years after YY, a bit like a sense of "how long a war can last".

The circuit alone reaches 72.7% probability difference, which is 89.5% of the full model's score. The mechanism also works in some other "greater-than" contexts. But it also fires in a few contexts where it should not. The authors call it "complex but general".

Grokking and modular addition: a network that adds with a clock

Grokking

In 2022, Power and colleagues found a strange training effect. They called it grokking. A small network learns a math task. First it memorizes the training examples: training accuracy is 100%, but test accuracy is poor. Then, after many more steps with no visible change, test accuracy suddenly jumps to near 100%. The network has suddenly generalized.

Why does this happen? Nanda and colleagues answered this question by fully reverse engineering one such network.

The task

The model is a one-layer transformer. The task is modular addition: given a and b, output (a + b) mod p, with p = 113. "mod p" means "wrap around after p", like hours on a clock: on a 12-hour clock, 9 + 5 = 2. The model trained on 30% of all possible (a, b) pairs, with weight decay. Weight decay is a training penalty that keeps weights small.

The algorithm: addition as rotation

The trained network uses an algorithm that nobody programmed. It is based on circles.

1. Embed on a circle. The embedding turns each number a into a point on a circle: (cos(w·a), sin(w·a)), with w = 2πk/p for some whole number k. The network does this for a few key frequencies k, about five of them.
2. Rotate. The attention and MLP layers combine the points for a and b. They use the trigonometry identities
   cos(w(a+b)) = cos(wa)·cos(wb) − sin(wa)·sin(wb)
   sin(w(a+b)) = sin(wa)·cos(wb) + cos(wa)·sin(wb)
   In circle terms: start at angle w·a, then rotate by angle w·b. You land at angle w·(a+b).
3. Read out. For each possible answer c, the unembedding computes cos(w(a+b−c)). This is largest (equal to 1) exactly when c = (a+b) mod p. Summing over the key frequencies makes the right answer stand out even more.

Try it. With one frequency, the right answer has the top logit, but its neighbours are almost as high. With five frequencies, the waves add up only at the right answer. Everywhere else they partly cancel. This is why the network uses several frequencies: it makes the answer sharp.

Three phases of training

Once they knew the algorithm, the authors could measure it during training. They defined progress measures: numbers that track the hidden circuit, not only the loss. For example, the restricted loss keeps only the key-frequency parts of the model. The excluded loss removes them. With these tools, they saw three phases.

1. Memorization. The network memorizes the training pairs. Train loss falls. Test loss stays high.
2. Circuit formation. The clock algorithm slowly forms. Train accuracy is already 100%, so the train loss shows nothing new. Test loss still stays high, because the memorized parts still dominate the output.
3. Cleanup. Weight decay removes the memorization parts, because they cost large weights. Now only the clean circuit remains. Test loss drops suddenly. This is the visible "grokking" moment.

The Clock and the Pizza: not always the same algorithm

Do all networks learn the clock? This follow-up said no. Small changes in hyperparameters and initialization gave different algorithms. The authors found a second algorithm and called it the pizza.

In the pizza algorithm, the network adds the two circle points as vectors and looks at their midpoint. The direction of the midpoint gives information about a + b. A nonlinearity is then needed to read the answer. Some networks even ran several algorithms in parallel. Networks with less flexible attention tended toward the pizza.

Othello-GPT: a world model from move lists

Do language models only learn surface statistics, or do they build a model of the world behind the text? This is a big debate. Othello-GPT gave a clean test case.

The setup

Othello is a board game on an 8×8 grid. Two players place black and white pieces. When you place a piece, you flip the opponent's pieces that lie in a line between your new piece and one of your old pieces.

The researchers trained a GPT model (8 layers, 512 dimensions, 8 heads) on millions of game transcripts. Each transcript is just a list of moves, like "D3, C5, F6, …". The model's only job was to predict a legal next move. It never saw the board. It never got the rules.

What they found

• The model predicts legal moves very well. To do that, it needs to know which squares are occupied and by whom. That is hard to do from the move list alone.
• They trained probes to read the state of each square (empty, black, white) from the activations. Linear probes worked poorly: an error rate of about 20% (20.4% at best). Nonlinear probes (small 2-layer MLPs) worked well: an error rate of about 1.7%. The authors concluded that the world model was real, but nonlinear.
• They also intervened. They used the probe's gradient to edit the activations so that the probe read a different board, for example with one square flipped. The model then predicted the legal moves of the edited board. So the board representation is not only present: the model uses it.

The follow-up: "mine" versus "theirs"

Why did linear probes fail? Neel Nanda had a guess. The model plays both sides, and the players take turns. So maybe the model does not care about "black" and "white". Maybe it cares about "my pieces" and "their pieces", relative to the player who moves next.

He was right. A linear probe for "mine / theirs / empty" reads the board with very high accuracy. A linear probe for black/white fails because the meaning of each direction flips every move. Try it below.

The follow-up also found features for "was this square just flipped?". Simple vector arithmetic along the mine/theirs directions (add or subtract the probe direction for a square) changed the model's predicted moves, as expected. Several edits at once worked less well. So the representation is linear and causal.

Factual recall: where do facts live?

A language model knows many facts: "The Eiffel Tower is in Paris", "Michael Jordan plays basketball". Where in the network are these facts stored, and how are they retrieved? Several studies attacked this question. Together they tell a story with a twist.

Step 0: MLPs as key-value memories

An MLP layer has two weight matrices. Geva and colleagues read them as a memory:

• Each row of the first matrix is a key. It detects a pattern in the input, such as "text about TV shows" or "a word that ends in -ing".
• Each matching row of the second matrix is a value. It pushes up some output words.

Keys in lower layers detect shallow patterns. Keys in upper layers detect topics and meaning. The layer output is a mix of many memories. Later layers refine the result.

Step 1: causal tracing and ROME

Meng and colleagues used a prompt like "The Space Needle is in downtown" (answer: Seattle). Then they used causal tracing, the noise-based form of activation patching from Chapter 5. In short: add noise to the subject tokens ("The Space Needle") so that the model no longer says Seattle, then restore one clean hidden state at a time and check how much of "Seattle" comes back.

Two places mattered most in GPT-2 XL. First: the MLPs in the middle layers, at the last token of the subject. Second: late layers at the final token, where attention moves the answer to the output.

The authors then built ROME (Rank-One Model Editing). They treated the middle MLP as a key-value store, as Geva suggested. They made a small "rank-one" change to its weights so that the key "Space Needle" maps to a new value, for example "Paris". After the edit, the model said "The Space Needle is in Paris" in many different phrasings. They tested edits on a dataset called CounterFact. A follow-up, MEMIT, edits thousands of facts at once.

Step 2: three stages of recall

Geva and colleagues blocked attention edges ("attention knockout") and projected states onto the vocabulary. They found three stages:

1. Subject enrichment. Early and middle MLPs add many attributes of the subject to the last subject token. After this step, the "Needle" position carries "Seattle", "tower", "Washington", and more.
2. Relation propagation. Information about the relation ("is in the city of") flows to the final position.
3. Attribute extraction. At the final position, attention heads look back at the enriched subject token and pull out the attribute that fits the relation.

Step 3: the twist — inside the lookup table

The DeepMind team asked Pythia 2.8B which sport about 1,500 athletes play, for example "Michael Jordan plays the sport of" → basketball. They confirmed the three-stage shape: attention layers 0–1 join the name tokens, MLPs 2 to 6 look up the fact on the last name token, and later attention heads extract it. The sport ends up as a linear direction in the residual stream.

Then they tried to explain how MLPs 2–6 do the lookup, neuron by neuron. They failed. The MLPs did not use a sparse, readable mechanism. The team's best description: the early MLPs "hash" the name tokens into a vector that is almost orthogonal to other names, and later neurons look up these hashed vectors. It acts like a big hash table: "this exact token combination → basketball". A lookup table has no short description. There may be no simple story to find.

A critique: where you edit is not where it is stored

ROME used causal tracing to choose the layer to edit. Hase and colleagues tested if that choice matters. It does not. You can edit a fact successfully in layers that causal tracing does not highlight. Which layer you edit predicts success much better than what causal tracing says about that layer.

Reusable mechanisms: successor heads, function vectors, binding IDs

The studies above each explain one task. A second wave of work in 2023 found mechanisms that the model reuses across many tasks. These look more like the "subroutines" of a program.

These three results share a theme. The model builds abstract, reusable representations: "item number n", "the task is antonyms", "tag number 2". Then it reuses them in many contexts. This is good news for interpretability. If models reuse parts, we can learn a part once and recognize it in many places.

Arithmetic without algorithms: a bag of heuristics

The modular addition network used a clean algorithm. Do large language models do arithmetic the same way? Nikankin and colleagues studied models such as Llama3-8B on problems like "226 − 68 =". The answer was no.

The models did not use a single clean algorithm. They also did not simply memorize answers. Instead, a sparse set of MLP neurons each implemented a simple heuristic, a rule of thumb. Examples of such rules:

• "Fire if the first operand is between 100 and 200."
• "Push up answers that end in 5."
• "Push up even answers."

Each rule is weak alone. But many rules vote together, and the votes add up to the correct answer. The authors call this a bag of heuristics. The mechanism appeared early in training and was the main source of accuracy.

Anthropic later saw a similar picture in Claude 3.5 Haiku with attribution graphs (diagrams of which internal features cause which): one path estimates the rough size of the sum, and another path gets the last digit exactly (Chapter 9). Models seem to prefer many parallel, approximate paths over one exact procedure.

Does circuit analysis scale? A 70B model

All the studies so far used small models. GPT-2 small has 124 million parameters. Does the method still work on a model 500 times larger? The DeepMind team tested this on Chinchilla 70B.

The task: a multiple-choice question with options labelled A, B, C, D. The model knows the right answer text. It must output the right letter.

• The standard tools — logit attribution, attention pattern viewing, activation patching — scaled naturally to 70B.
• They found a small group of "correct letter heads". These heads attend from the answer position to the letter of the correct option, and copy that letter to the output.
• Inside these heads, they partly found an "Nth item in a list" feature. But when they randomized the labels, the explanation was incomplete.

Universality: do different models learn the same circuits?

The third claim of "Zoom In" (Chapter 1) was universality: similar features and circuits form in different models. If true, research on one model transfers to others. What does the evidence say?

Gurnee and colleagues trained five GPT-2 models that were identical except for the random seed. They ran 100 million tokens through all five and compared every neuron's activations. A neuron is "universal" if a neuron in every other model fires on the same inputs.

Only 1–5% of neurons were universal. But those neurons were special. They were usually interpretable, and they fell into families: neurons for single tokens, for letters of the alphabet, for position, for syntax and for meaning. Some had clear jobs:

• Attention deactivation neurons switch off certain attention heads.
• Entropy neurons change how confident the next-token distribution is.
• Prediction and suppression neurons push up, or push down, a whole set of tokens.

What the case studies teach us

The case studies from 2022 to 2024 changed how people think about neural networks. Here are the main lessons.

1 · Circuits are real

Before these studies, many people thought that large networks were hopeless to read. The case studies showed real, testable algorithms: name copying, clock arithmetic, board tracking, copy suppression. Each claim names specific components, and each claim survived causal tests. Neural networks are not random soup.

2 · But circuits are messy

The clean stories have rough edges. IOI was 87% faithful and not fully complete. Greater-than explained 89.5%. The MLPs in fact recall looked like lookup tables. LLM arithmetic was a bag of heuristics, not one algorithm. Models seem to like many parallel, partly redundant paths.

3 · Ablation can mislead

Backup heads and self-repair mean that switching off one part often changes little. The model compensates. So a small effect after ablation does not prove a part is unimportant. Researchers now prefer careful patching, and they compare several kinds of ablation (Chapter 5).

4 · Universality is partial

Some mechanisms, like induction heads and successor heads, appear everywhere. Others depend on the seed and the settings. We cannot assume that a circuit from one model exists in another. We must check.

5 · Manual work does not scale

Each study took a team weeks or months, for one narrow behaviour, usually in a small model. A frontier model does thousands of behaviours. This scale problem drove the next phase of the field:

• Automated circuit discovery, such as ACDC and attribution patching (Chapter 5).
• Better units of analysis than neurons and heads: sparse autoencoder features (Chapter 8).
• Attribution graphs that trace circuits through those features for any prompt (Chapter 9).

References

1. Wang, Variengien, Conmy, Shlegeris, Steinhardt (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
2. McDougall, Conmy, Rushing, McGrath, Nanda (2023). Copy Suppression: Comprehensively Understanding an Attention Head. BlackboxNLP 2024 (published as "Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads").
3. Hanna, Liu, Variengien (2023). How does GPT-2 compute greater-than? NeurIPS 2023.
4. Power, Burda, Edwards, Babuschkin, Misra (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets.
5. Nanda, Chan, Lieberum, Smith, Steinhardt (2023). Progress measures for grokking via mechanistic interpretability. ICLR 2023.
6. Zhong, Liu, Tegmark, Andreas (2023). The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. NeurIPS 2023.
7. Chughtai, Chan, Nanda (2023). A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations.
8. Li, Hopkins, Bau, Viégas, Pfister, Wattenberg (2022). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023.
9. Nanda, Lee, Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. BlackboxNLP 2023.
10. Geva, Schuster, Berant, Levy (2021). Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021.
11. Meng, Bau, Andonian, Belinkov (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. Site: rome.baulab.info.
12. Geva, Bastings, Filippova, Globerson (2023). Dissecting Recall of Factual Associations in Auto-Regressive Language Models. EMNLP 2023.
13. Nanda, Rajamanoharan, Kramár, Shah (2023). Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level. Alignment Forum.
14. Hase, Bansal, Kim, Ghandeharioun (2023). Does Localization Inform Editing? NeurIPS 2023.
15. Gould, Ong, Ogden, Conmy (2023). Successor Heads: Recurring, Interpretable Attention Heads In The Wild. ICLR 2024.
16. Todd, Li, Sen Sharma, Mueller, Wallace, Bau (2023). Function Vectors in Large Language Models. ICLR 2024.
17. Feng, Steinhardt (2023). How do Language Models Bind Entities in Context? ICLR 2024.
18. Nikankin, Reusch, Mueller, Belinkov (2024). Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics. ICLR 2025.
19. Lieberum, Rahtz, Kramár, Nanda, Irving, Shah, Mikulik (2023). Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla.
20. Gurnee, Horsley, Guo, Kheirkhah, Sun, Hathaway, Nanda, Bertsimas (2024). Universal Neurons in GPT2 Language Models.
21. Miller, Chughtai, Saunders (2024). Transformer Circuit Faithfulness Metrics are not Robust. COLM 2024.