What is mechanistic interpretability?

We train neural networks. We do not write them. This chapter explains why that is a problem, and what it means to "reverse engineer" a network.

The problem: we build what we cannot read

A normal computer program has source code. A person wrote each line. If the program does something wrong, you can read the code and find the bad line.

A neural network is different. Engineers write the training code. Then training changes millions or billions of numbers, called weights. At the end, the weights do the task. But no person chose them, and no person knows what each one does.

Large language models now write code, answer medical questions, and act as agents. We know that they work on many tests. We do not know how they work inside. This gap is the central problem of the field.

Neural networks in five minutes

You need only a few ideas to follow this survey. We explain them here with simple words.

A neuron

A neuron takes some input numbers. It multiplies each input by a weight. It adds the results together, plus a constant called the bias. Then it applies a simple function. The most common function is ReLU: keep the number if it is positive, else output zero.

The output of the neuron is its activation. Try the widget below. Move the inputs and see how the activation changes.

Layers and activations

A layer is many neurons side by side. A network is many layers in a row. The first layer reads the input (pixels or words). Each next layer reads the activations of the layer before it. The last layer gives the output.

For one input, each layer produces a list of numbers: its activations. A list of numbers is a vector. If a layer has 512 neurons, its activation is a point in a space with 512 dimensions. Much of this survey is about the geometry of that space: which directions mean what.

Training

Training shows the network many examples. For each example, it measures the error. Then it moves every weight a little to make the error smaller. It uses the gradient to find the right direction. After millions of steps, the network does the task. But training only optimizes the error. It does not care if the result is easy for humans to read.

Transformers

Modern language models are transformers. They read text as tokens (words or parts of words). Each token has a vector that flows through the model. This vector is called the residual stream. Two kinds of blocks read and write it: attention heads move information between tokens, and MLPs process each token on its own. Chapter 4 explains this in detail.

What "mechanistic" means

Interpretability is a large field. Many methods explain what a model does, or which input parts mattered. Mechanistic interpretability asks a stronger question: what algorithm runs inside, step by step?

The term "mechanistic interpretability" comes from Chris Olah and his collaborators. It became common from about 2020. The line between the approaches is not sharp. Mech interp uses probes and attributions as tools. But its goal is a causal, step-by-step story that you can check.

The three core objects: features, circuits, universality

In 2020, Chris Olah and colleagues wrote "Zoom In: An Introduction to Circuits". It made three claims. These claims are still the base of the field.

The authors called these claims "speculative". Later work refined the first claim. Most features found so far are directions, but some are not single directions. For example, some language models store days of the week and months on a circle in a small subspace (Engels et al., 2024).

The story would be simple if each neuron were one feature. Sadly, this is often false. Many neurons respond to several unrelated things. This is called polysemanticity. The main explanation is superposition: the model stores more features than it has neurons. Much of the modern field is a fight with superposition. Chapter 3 explains it, and Chapter 8 shows the main tool to undo it.

The toolbox: observe, then intervene

Mech interp methods fall into two groups.

• Observational methods look at the model without changing it. They answer "what is represented here?" Examples: feature visualization, probes, sparse autoencoders.
• Interventional methods change something inside the model and watch the output. They answer "does this part cause the behaviour?" Examples: ablation, activation patching, steering.

Good research uses both. Observation gives a hypothesis. Intervention tests it. A probe can find information that the model never uses. Only an intervention shows that the model uses it.

Why does this matter?

People study mech interp for several reasons.

1. Safety. A model could look good on tests but have hidden goals or unsafe knowledge. If we can read its internal reasoning, we can find problems that tests miss. This is the main reason for much of the funding in the field.
2. Trust and debugging. If a model fails, we want to know why. Knowing the mechanism helps us fix the cause, not only the symptom.
3. Science. Neural networks learn real structure from data. Reading them may teach us new facts about language, vision, or even biology. It is also a new kind of science: we have full access to every weight, unlike in neuroscience.
4. Control. If we know which direction means "refuse" or "Golden Gate Bridge", we can turn it up or down. This gives new ways to steer models.

A useful comparison: neuroscience

Neuroscientists study brains. They record neurons, damage regions, and stimulate cells. Mech interp uses the same moves: we record activations, ablate components, and add steering vectors. But we have three big advantages.

• We can read every neuron and every weight, with no noise.
• We can run the same input again and get exactly the same result.
• We can change any part and see the effect at once. No ethics board is needed.

Chris Olah listed advantages like these in a 2021 note, "Interpretability vs Neuroscience". He argued that researchers should use them "as ruthlessly as possible". Still, the problems are hard. A large model has billions of parameters, and the features we care about are often spread across many neurons.

How to read this survey

The chapters follow the history of the field. Each chapter starts with a short summary and ends with a quiz. Underlined words show a definition when you hover over them. Boxes with an orange dot are interactive.