Causal tools: ablation and patching

Looking at activations shows what is correlated with a behaviour. To find what causes it, you must reach into the model, change one thing, and watch the output. This chapter explains how.

Why correlation is not enough

Chapter 4 gave us stories about heads and circuits, such as the induction head. How do we test such a story in a real model? This chapter gives the tools, which the field built mostly from 2020 to 2024. First, why looking is not enough. In Chapter 7 we meet probes. A probe reads information from activations. But a probe can find information that the model never uses. A neuron can fire on French text and still have no effect on the output. To show that a part matters, you must change it and see the output move.

This idea is old. Neuroscientists damage a brain region and watch which skill is lost. Doctors run trials where they change one thing and keep the rest fixed. In a neural network, this is much easier. We can change any number we like, and we can run the same input again and again with no noise.

Every method in this chapter follows the same three steps:

1. Pick a part. This can be a neuron, an attention head, the residual stream at one token, an edge between two parts, or a direction in activation space.
2. Replace its value. Use zero, an average, a value from a different input, or a value with noise added.
3. Measure the change. Look at how much the output moves.

The methods differ in three ways: what value they put in, what unit they change (a node, an edge, or a subspace), and how they make it fast (real reruns or gradients).

Ablation: turn it off and see what breaks

The simplest intervention is an ablation. You remove the output of one part and measure how much worse the model gets. But there is a hard question. What does "remove" mean? During training, a part never outputs "nothing". It always outputs some number. So you must choose a replacement value. There are four common choices.

The choice really matters. Some parts of GPT-2 small output a large, almost constant vector. This vector acts like a bias. If you set it to zero, the model breaks, even though the part carries no information about the input. Mean ablation shows the truth: the part is not important for the task. The IOI paper (Chapter 6) used mean ablation over an "ABC" dataset of prompts with three different names. The ACDC authors found that zero ablation and resample ablation can give different circuits.

Necessary or sufficient?

Ablation of one part tests whether that part is necessary: does the model still work without it? You can also ablate everything except a proposed circuit. This tests whether the circuit is sufficient: is it enough on its own? A good circuit claim needs both tests. Wang et al. (2022) named three criteria for circuits:

• Faithfulness: the circuit alone does the task about as well as the full model.
• Completeness: the circuit contains all the parts that matter.
• Minimality: the circuit contains no parts that do not matter.

The root idea: causal mediation analysis

Vig and colleagues gave the field its first clear causal recipe. They used a prompt like "The nurse said that". Then they compared the probability of "she" and "he". They asked: which internal parts carry the effect of the word "nurse"?

Statistics has words for this. A mediator is a variable that sits between a cause and an effect.

• The total effect is how much the he/she ratio changes when you change the input word (for example, "nurse" to "man").
• The indirect effect of one component: keep the original input, but set that component to the value it would have under the changed input. If the output moves a lot, the component carries the effect.
• The direct effect: change the input, but freeze the component at its original value.

They found that the gender effect was sparse. A small number of attention heads in middle layers, plus some neurons, carried most of it. The effect was also synergistic (some parts amplify it, others damp it) and split into a direct part and a part that flows through the mediators.

Activation patching

Activation patching is the main workhorse of mech interp. Three communities found it at about the same time, so it has three names:

• Interchange intervention — from Atticus Geiger and colleagues at Stanford (2020 onward).
• Causal tracing — from the ROME paper by Meng et al. (2022). It uses noise.
• Activation patching — the name used by Anthropic, Redwood Research and Neel Nanda from 2022.

The three runs

We use the IOI task as the example. IOI means "indirect object identification". The model must finish a sentence with the right name. Chapter 6 tells the full story of the circuit that GPT-2 small uses. In short: some heads notice which name is repeated (S-inhibition heads pass this signal on), and other heads copy the other name to the output (name mover heads). We will meet these names below.

Denoising and noising

You can patch in two directions. They answer different questions. Heimersheim and Nanda (2024) gave them clear names.

• Denoising: run the corrupted input and patch in a clean activation. This asks: "Is this part sufficient to bring the behaviour back?"
• Noising: run the clean input and patch in a corrupted activation. This asks: "Is this part necessary? Does breaking it break the behaviour?"

What to measure

You need one number that says "how much the model prefers the right answer". There are several choices.

• Logit difference (the usual choice): logit(" Mary") − logit(" John"). It is linear in the final residual stream, so it splits cleanly into parts. It also cancels shifts that affect both names the same way.
• Probability of the right answer: easy to understand, but it saturates. Softmax squashes large changes. It can hide parts that push against the answer.
• KL divergence (a standard measure of how different two probability distributions are) between the patched and clean output distributions: it does not need a specific answer. ACDC uses it.

Most heatmaps show a normalized score:

A score of 0 means "patching did nothing; the output is still corrupted". A score of 1 means "fully restored to clean". Scores can be below 0 or above 1.

Try it: real activation patching on GPT-2 small

The widget below uses real numbers. We computed them for this survey. We ran GPT-2 small (12 layers, 12 heads per layer) with the Hugging Face transformers library. We used denoising: run the corrupted prompt, paste in one clean activation, and read the normalized logit difference. On the clean prompt, the logit difference is +2.01 (the model prefers " Mary"). The scores come from one prompt pair, so they are an example, not an average.

Here is what to look for.

• With the S2 swap, only the second name differs between the two prompts. So in layer 0, only the S2 column (the second " John") can matter. The signal stays at S2 through the early layers. Around layers 7 to 9, it moves to the final token " to". From layer 9 on, almost only the final column is bright. The information has been moved by attention heads from S2 to the end. This "hand-off" shape is the classic IOI result.
• With the "replace all names" corruption, the corrupted prompt is "When Alice and Bob went to the store, Bob gave a drink to". All three name tokens differ, but the pattern (one name repeated) stays the same. Note: this is not the "ABC" dataset of the IOI paper, where three different names appear. Now the " Mary" column (the indirect object) is very bright in early layers. It is the source of the answer. The S1 column is negative: restoring the first " John" alone confuses the model. Late layers again focus on the final token.
• Switch to Attention out and MLP out to see which kind of block does the work at each step.

Zooming in: one cell per attention head

The heatmap above tells us where. Now we ask who. We patch the output of each of the 144 attention heads, one at a time. Hover a head to see the role that the IOI paper gave it.

The two corruptions light up different heads, because they test different information.

• The S2 swap only changes which name is repeated. So it lights up the heads that carry the "this name is repeated" signal: the S-inhibition heads 7.3, 7.9, 8.6 and 8.10. Head 9.9, a name mover, also shows up. With "All positions", the induction head 5.5 (which acts at S2) also lights up.
• The replace-all-names corruption keeps the repeat pattern but changes the names themselves. So the S-inhibition signal is the same in both runs, and it lights up the heads that copy a name to the output: the name mover heads 9.6, 9.9 and 10.0. Some backup name movers (10.1, 10.10, 11.2) are also positive.
• In both cases, head 10.7 is strongly negative, and with replaced names 11.10 is too. These are the negative name movers. They push against the answer. (With replaced names, the backup heads 10.2 and 10.6 are also negative on this one prompt, so a "backup" label does not fix the sign of a head.) Later work showed that 10.7 does "copy suppression" (Chapter 6).

Best practices

Zhang and Nanda tested the choices one by one. They found:

1. How you corrupt matters. Gaussian noise on the embeddings (the ROME method) can push the model off-distribution. It can point to different layers than a natural corruption. They recommend symmetric token replacement: swap in a similar token, such as another name or another city.
2. The metric matters. Probability can miss negative components, such as the negative name movers. Logit difference shows them.
3. Patching many layers at once (a "sliding window", as in ROME) can make an effect look more localized than it is.

Their main advice: start coarse, then refine. First patch the residual stream by layer and position. Then split into attention and MLP. Then go to single heads. Then use path patching to test connections.

Causal tracing: where are facts stored?

Meng and colleagues asked where a model stores facts. They used prompts like "The Space Needle is in downtown" → "Seattle". Their version of patching is called causal tracing.

1. Clean run: the model says "Seattle".
2. Corrupted run: add Gaussian noise to the embeddings of the subject tokens ("The Space Needle"). The noise had a standard deviation of 3 times the normal spread of embeddings. Now the model usually cannot recall "Seattle".
3. Restore one clean hidden state at one (token, layer). Measure how much the probability of "Seattle" comes back. They did this for the residual stream, for MLP outputs and for attention outputs.

They read the early site as: "middle-layer MLPs store the fact, keyed by the last token of the subject." Then they built ROME (Rank-One Model Editing). ROME changes one MLP weight matrix with a small, rank-one update to change one fact, for example to make the model say the Space Needle is in Paris. They used GPT-2 XL (1.5B parameters) and GPT-J.

Interchange interventions and causal abstraction

Atticus Geiger, Christopher Potts, Thomas Icard and colleagues built a formal theory around patching. They call it causal abstraction.

The idea works like this.

1. Write a small high-level program with named variables. For the task "is (a + b) > c?", the program has a variable S = a + b, and then the output is "S > c".
2. Guess which neurons in the network hold the variable S.
3. Take two inputs. Run the network on input 1, but swap in the "S neurons" from input 2. This swap is an interchange intervention.
4. Ask the program what should happen if S had input 2's value. Compare with what the network did.

If the network matches the program on most swaps, then the program is a good description of the network. The fraction of matches is called interchange intervention accuracy (IIA). This gives a clear, testable meaning to "the network implements this algorithm".

The same group later used these swaps as a training signal (Interchange Intervention Training, ICML 2022), and wrote a full theory paper, "Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability" (2023).

Path patching: test a connection, not a node

Activation patching asks "does this node matter?". But a circuit is made of connections. Path patching asks "does the connection from node A to node B matter?".

Here is the trick. Patch A's clean output, but feed it only into B. Every other part after A still sees A's corrupted value. This isolates one edge.

The IOI team used path patching backwards from the output. First they found the heads that write the answer directly (the name movers). Then they asked which heads feed the name movers' queries (the S-inhibition heads). Then which heads feed those (duplicate token and induction heads). Step by step, they built the full circuit. Chapter 6 tells the story.

Causal scrubbing: test a whole hypothesis

Redwood Research wanted a strict test for a full explanation. Their method is called causal scrubbing. It works like this.

1. Write your hypothesis as a map. Each model part maps to a step in a simple computation. Also say what information each part depends on. For example: "this head only uses the previous token".
2. For each part, find other inputs that your hypothesis says are equivalent. For the example: any input with the same previous token.
3. Replace the part's activation with its value on one of those equivalent inputs. Do this for everything at once. This "scrubs" away all information your hypothesis says does not matter.
4. If the hypothesis is right, the model should still perform well.

The score is the fraction of loss recovered: how much of the gap between the full model and a fully scrambled model your hypothesis keeps. They tested it on induction heads in a small attention-only model and on a model that checks if parentheses are balanced.

ACDC: automatic circuit discovery

Finding the IOI circuit by hand took months. ACDC (Automatic Circuit DisCovery) does the search automatically.

1. Treat the model as a graph. Nodes are heads and MLPs. Edges connect them.
2. Start at the output and move backwards. For each edge, replace its value with the value from the corrupted input.
3. Measure the KL divergence between the new output and the original output. If it goes up by less than a threshold τ, delete the edge for good. Otherwise, keep it.
4. The edges that remain are the circuit.

ACDC found the circuits that people had found by hand: IOI, greater-than, docstring, induction, and Tracr programs. For greater-than in GPT-2 small, it kept 68 of about 32,000 edges, and all of them had also been found manually.

Attribution patching: patching at the speed of gradients

Activation patching needs one forward pass for every cell in the heatmap. For a large model with millions of possible patches, this is far too slow. Attribution patching uses a shortcut from calculus.

The idea: if you change an activation a by a small amount, the metric changes by about the gradient times that amount. So:

You need one clean forward pass, one corrupted forward pass and one backward pass. Then you get an estimate for every component at once. This is a first-order Taylor approximation: it replaces the true curve by a straight line.

The family of fast methods

Edge Attribution Patching (EAP) scores each edge u → v as (clean output of u − corrupted output of u) · (gradient of the metric with respect to the input of v). Two forward passes and one backward pass give scores for all edges. Keep the top edges.

EAP-IG averages the gradient along the straight line from corrupted to clean activations (often 5 to 50 steps). This fixes much of the error of the straight-line estimate. The authors also argued that we should judge a circuit by faithfulness (does it reproduce the model's behaviour?), not by how much it overlaps with a "true" circuit. EAP-IG became a strong default method.

The DeepMind team found that attribution patching (AtP) is the best fast method, but it has two kinds of false negatives (important parts that it scores as unimportant):

1. Saturated attention. When the softmax is saturated, its gradient is tiny. The fix: for queries and keys, really recompute the attention pattern, and only then use the straight-line estimate.
2. Cancellation. A part's direct effect and indirect effect can cancel. The fix, called GradDrop: switch off the gradient through one layer at a time and average the results.

They tested it on Pythia models up to 12B parameters.

Distributed Alignment Search: patch directions, not neurons

Because of superposition, a variable may not live in single neurons. It may live in a direction or a subspace that mixes many neurons. Patching single neurons will then miss it.

DAS learns a rotation of activation space. The model's weights stay frozen. Only the rotation is trained. The training goal: when you do interchange interventions on the first k rotated dimensions, the network should act like the high-level program. In other words, DAS searches for the subspace that holds the variable.

Boundless DAS learns the size of the subspace with soft masks, so it scales to a 7-billion-parameter model. The task was "Price Tagging": "Please say yes only if it costs between [1.30] and [8.55] dollars, otherwise no." They found that Alpaca (7B) uses two clear boolean variables: "is the price above the lower bound?" and "is the price below the upper bound?". This held even when the inputs and instructions changed.

Pitfalls: when interventions mislead

Interventions are stronger than observation. But they can still mislead. Here are the main traps.

Testing the testers: ground truth and benchmarks

How do we know if a circuit-finding method works? In a normal model, we do not know the true circuit. So researchers built models where they do know it.

Tracr takes programs written in RASP, a small language that describes what transformers can do (Weiss, Goldberg and Yahav, 2021, "Thinking Like Transformers"). It compiles them straight into transformer weights. Examples include sorting, counting tokens, reversing a list, and checking balanced brackets. Because a person wrote the program, we know the true circuit. Limit: compiled models are not trained, so they look unnatural. For example, they have no superposition unless you add a compression step.

InterpBench trains small transformers with "Strict Interchange Intervention Training" (SIIT). Each model is trained to implement a known circuit, but with realistic trained weights. The authors used them to compare ACDC, EAP and other methods.

MIB gives the field a shared test. It has two tracks:

• Circuit localization: find the components and edges that matter. Scored by faithfulness across circuit sizes.
• Causal variable localization: find a feature in a hidden vector that matches a task variable. Scored with interchange intervention accuracy.

The main results: attribution methods (such as EAP-IG) and mask-learning methods (which learn which edges to keep) did best at circuit localization. Supervised DAS did best at causal variable localization. And, surprisingly, features from sparse autoencoders were not better than plain neurons on that track. (A sparse autoencoder, or SAE, splits an activation into many readable feature directions.) (Chapter 8 discusses this debate.)

Direct logit attribution: a cheap first look

Some questions do not need any intervention. In a transformer, the final residual stream is a sum: the embeddings plus the output of every attention head and every MLP (Chapter 4). So we can split the final logit difference into one piece per component.

The recipe: take each component's output at the final position. Apply the final LayerNorm scale (frozen from the full run). Then take the dot product with the difference of unembedding vectors, W_U[" Mary"] − W_U[" John"]. The result is that component's direct effect on the logit difference. All the pieces add up (about) to the total.

A note on gradient attribution

Older attribution methods also exist for language models. Gradient × input scores each input token by the gradient times its embedding. Integrated Gradients (Sundararajan, Taly and Yan, 2017) integrates gradients along a path from a baseline, so the scores add up to the full output difference. Raw attention weights are not reliable explanations; this was the subject of the 2019 debate "Attention is not Explanation" (Jain and Wallace) and its reply (Wiegreffe and Pinter). Newer decomposition methods, such as information flow routes (Ferrando and Voita, 2024), build a graph of important edges in one forward pass. All of these methods have the same weak points: gradients are local, they can be saturated or noisy, and the baseline is an arbitrary choice.

All methods side by side

A practical recipe

Here is a checklist for a causal study of one behaviour. It follows the advice of Heimersheim & Nanda and Zhang & Nanda.

1. Define the behaviour with many prompts, not one. Make sure the model does it reliably.
2. Pick a metric. Prefer logit difference between a right and a wrong answer.
3. Design the corruption. Change only the thing you care about. Prefer natural token swaps over noise. Keep the token count the same.
4. Start coarse. Patch the residual stream by layer and position.
5. Refine. Split into attention and MLP outputs, then single heads. Use attribution patching to screen many parts quickly, then check the top ones with real patching.
6. Test connections with path patching, or run ACDC / EAP-IG to propose a circuit.
7. Test the circuit: faithfulness (ablate everything else), completeness and minimality. Use mean or resample ablation, not zero.
8. Look for traps: try both noising and denoising; watch for self-repair and negative components; check that results hold across prompts.
9. Explain the parts. Look at attention patterns, direct logit attribution, and what features the parts read and write.

References

1. Vig, J., Gehrmann, S., Belinkov, Y., et al. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS.
2. Geiger, A., Lu, H., Icard, T., Potts, C. (2021). Causal Abstractions of Neural Networks. NeurIPS.
3. Geiger, A., Wu, Z., Lu, H., et al. (2022). Inducing Causal Structure for Interpretable Neural Networks. ICML.
4. Geiger, A., et al. (2023). Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability.
5. Meng, K., Bau, D., Andonian, A., Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS.
6. Hase, P., Bansal, M., Kim, B., Ghandeharioun, A. (2023). Does Localization Inform Editing? NeurIPS.
7. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
8. Goldowsky-Dill, N., MacLeod, C., Sato, L., Arora, A. (2023). Localizing Model Behavior with Path Patching.
9. Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., et al. (2022). Causal Scrubbing. Redwood Research.
10. Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., Garriga-Alonso, A. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS.
11. Nanda, N. (2023). Attribution Patching: Activation Patching At Industrial Scale.
12. Syed, A., Rager, C., Conmy, A. (2023). Attribution Patching Outperforms Automated Circuit Discovery.
13. Hanna, M., Pezzelle, S., Belinkov, Y. (2024). Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. COLM.
14. Kramár, J., Lieberum, T., Shah, R., Nanda, N. (2024). AtP*: An efficient and scalable method for localizing LLM behaviour to components.
15. Zhang, F., Nanda, N. (2023). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024.
16. Heimersheim, S., Nanda, N. (2024). How to use and interpret activation patching.
17. Geiger, A., Wu, Z., Potts, C., Icard, T., Goodman, N. D. (2023). Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. CLeaR 2024.
18. Wu, Z., Geiger, A., Icard, T., Potts, C., Goodman, N. D. (2023). Interpretability at Scale: Identifying Causal Mechanisms in Alpaca. NeurIPS.
19. Makelov, A., Lange, G., Nanda, N. (2023). Is This the Subspace You Are Looking For? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.
20. Wu, Z., Geiger, A., et al. (2024). A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments.
21. McGrath, T., Rahtz, M., Kramár, J., Mikulik, V., Legg, S. (2023). The Hydra Effect: Emergent Self-repair in Language Model Computations.
22. Rushing, C., Nanda, N. (2024). Explorations of Self-Repair in Language Models. ICML.
23. Li, M., Janson, L. (2024). Optimal ablation for interpretability. NeurIPS.
24. Lindner, D., Kramár, J., Farquhar, S., et al. (2023). Tracr: Compiled Transformers as a Laboratory for Interpretability. NeurIPS.
25. Weiss, G., Goldberg, Y., Yahav, E. (2021). Thinking Like Transformers. ICML.
26. Gupta, R., Arcuschin, I., Kwa, T., Garriga-Alonso, A. (2024). InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques. NeurIPS Datasets & Benchmarks.
27. Mueller, A., Geiger, A., Wiegreffe, S., et al. (2025). MIB: A Mechanistic Interpretability Benchmark. ICML.
28. Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic.
29. Sundararajan, M., Taly, A., Yan, Q. (2017). Axiomatic Attribution for Deep Networks (Integrated Gradients). ICML.
30. Jain, S., Wallace, B. C. (2019). Attention is not Explanation. NAACL.
31. Ferrando, J., Voita, E. (2024). Information Flow Routes: Automatically Interpreting Language Models at Scale. EMNLP.
32. Bhaskar, A., Wettig, A., Friedman, D., Chen, D. (2024). Finding Transformer Circuits with Edge Pruning. NeurIPS.
33. Miller, J., Chughtai, B., Saunders, W. (2024). Transformer Circuit Faithfulness Metrics are not Robust. COLM.
34. Méloux, M., Maniu, S., Portet, F., Peyrard, M. (2025). Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? ICLR.