Probing and steering

If concepts live as directions inside a model, we can do three things with them. We can read them with a probe. We can erase them to test if the model needs them. And we can write them to change what the model does. This chapter covers all three, from 2016 to 2026.

Reading versus writing a model's mind

Chapters 5 and 6 traced whole circuits, part by part. This chapter follows a simpler line of work that ran in parallel, from 2016 to 2026. Its idea: treat one concept as one direction, then read it or write it. A language model turns each token into a long vector at every layer. In a transformer, this vector is the residual stream. This chapter is about three ways to work with these vectors.

1. Probing (reading). Train a small, simple model to predict some property X from the activations. If it works, information about X is in the activations.
2. Erasure (checking use). Remove the part of the activations that carries X. Then see if the model's behaviour changes. If it does, the model uses X.
3. Steering (writing). Add a vector to the activations while the model runs. This pushes its behaviour in one direction: more positive, more honest, more refusing.

All three rely on one idea: the linear representation hypothesis. It says that many concepts are stored as directions in activation space. If "truth" is a direction, you can read it with a dot product. You can delete it with a projection. You can push it with an addition.

"The information is there" is not "the model uses it"

This is the most important idea in this chapter. Read it slowly.

A probe answers one question: could someone who looks at these numbers guess X? It does not answer a second question: does the model itself look at X when it makes its next prediction?

A probe can mislead you in two ways.

1. The probe does the work. A powerful probe, such as a big MLP, can learn the task by itself from almost any rich input. Then high accuracy tells you about the probe, not about the model.
2. The information is present but unused. The information can be there as a side effect. It can correlate with something that the model does use. But the model may never read it.

The fix is a causal intervention. Change the activations along the probed direction. Then watch the output. If removing "is this a noun?" makes the model worse at predicting nouns, the model used that information. If adding a "truth" direction makes the model call false statements true, the direction is linked to the behaviour.

Classic probing, 2016 to 2019

Probing started as a way to ask "what does each layer know?" The method is simple. Freeze the trained network. Collect activations at one layer for many inputs. Train a small classifier on those activations to predict a label. Repeat for each layer.

Alain and Bengio tested image classifiers, such as convolutional networks on MNIST and Inception v3 on ImageNet. Accuracy went up steadily from layer to layer. They also found a strange case. In a very deep model with a skip connection, the middle layers were "dead". The probes showed no gain there. So probes can also find bugs.

Conneau and colleagues found that sentence encoders capture many of these properties. The training objective and the architecture both matter. They also found an early warning sign: some properties could be read even from untrained, random encoders. A good probe result needs a good baseline.

Old NLP systems were pipelines. First a tagger found the part of speech (noun, verb). Then a parser found the sentence structure. Then later stages found meaning, such as "who did what to whom", and which words refer to the same thing. Tenney and colleagues computed, for each task, the expected layer where BERT solves it. The order was roughly: part of speech, then constituents, then dependencies, then semantic roles, then coreference. The model could also revise early guesses in later layers. The differences between tasks were small, so "pipeline" is a loose analogy.

The structural probe: a whole tree inside the geometry

Most probes predict a label. Hewitt and Manning asked a bigger question. Does the model store a whole syntax tree? A dependency parse tree links each word to the word it depends on. The "tree distance" between two words is the number of links between them.

They learned one matrix B. They then measured the squared distance ‖B(hᵢ − hⱼ)‖² between two transformed word vectors. Training made this distance match the tree distance. If a single linear map can do this for every sentence, the tree is "drawn" into the geometry of the vectors.

It worked well. The paper reports results on the Penn Treebank test set. "UUAS" is the percent of tree links that the probe recovers. The best result was BERT-large layer 15, with UUAS 82.5. Baselines were much lower. A probe on ELMo's non-contextual layer 0 scored 26.8. A simple left-to-right chain scored 48.9. A random contextual projection scored 59.8. The output size of B did not need to be large. Above about 64 to 128 dimensions there was no more gain. So syntax lives in a low-rank subspace: a small slice of the full space.

Is the probe cheating? Control tasks and MDL

By 2019, probe results were everywhere. Researchers started to ask a hard question. When a probe gets 97% accuracy, who did the work: the model or the probe?

A control task gives each word type a random, but fixed, label. For example, every "cat" gets label 7 and every "the" gets label 3. This label means nothing about language. The only way to solve it is to memorize which word is which. A probe that does well on the control task is a good memorizer.

High selectivity means the probe reads real structure, not word identity. Hewitt and Liang found that MLP probes reach high accuracy on control tasks. They memorize. So their high accuracy on the real task means less. Linear probes were more selective. Dropout did not reliably fix MLP probes. Other limits helped: smaller hidden layers, less probe training data and weight decay.

Try it yourself below. The points are made-up activations for two classes. The probe is a direction. Drag in the plot, or use the slider, to turn it. Then switch to random labels (a control task), and switch the probe type.

What should you see? With real labels, a good direction reaches high accuracy. With random labels, no direction does much better than a coin. The huge "MLP" in this demo is a perfect memorizer (it copies the label of the nearest point). It gets 100% on both tasks. Its selectivity is zero. So its 100% on the real task tells you nothing about the model.

Two other answers: information theory and MDL

Pimentel and colleagues took the opposite view. A probe estimates how much mutual information the representation has about the property. A weaker probe only gives a looser estimate. So, they said, use the best probe you can. They also made a sharp point. A contextual embedding is computed from the sentence. So it cannot contain more information about the sentence than the sentence itself. The useful question is the gain over a baseline, such as simple word embeddings. In their tests, BERT gave only modest gains for part of speech and dependency labels.

These two views do not really fight. They ask different questions. "Is the information there at all?" needs a strong probe. "Is it there in an easy, linear form that a model could use?" needs a simple probe. Mech interp usually cares about the second question.

Voita and Titov replaced accuracy with description length. Imagine that you must send all the labels to a friend who already has the representations. If the labels are easy to predict from the representations, the message is short. Their "online code" trains the probe on a growing amount of data. It pays a cost for each new block of labels. A probe that learns fast from few examples gives a short code. This measure captures both quality and effort. It separated real representations from random ones, and real tasks from control tasks, much more clearly than accuracy did. Accuracy differences were often tiny while code lengths differed a lot.

Is the information actually used? Erasure methods

Control tasks and MDL ask whether the probe cheated. A second problem remains. Even an honest probe only shows that information is present. To test use, researchers learned to remove the information and watch what breaks.

INLP works in rounds. In each round, it trains a linear classifier for the concept, for example gender. The classifier defines a direction. INLP then projects every vector onto the nullspace of the classifier. This removes that direction. Then it trains a new classifier and repeats. It stops when no linear classifier can beat chance. The first use was fairness: removing gender from word embeddings. But it is also a tool for causal tests.

The main result: probe accuracy and actual use did not line up. Some properties were easy to probe but had little effect when removed. Others, like part of speech and dependency information, did matter for BERT's masked-word prediction. The comparison with "remove random directions" is important. Any removal hurts a little. Only a removal that hurts more than random shows use.

INLP can remove more than needed, because it runs many rounds. LEACE (LEAst-squares Concept Erasure) solves the problem in one step. The key condition is simple: after erasure, the average vector of each class must be the same. If the class means are equal, no linear classifier can tell the classes apart. LEACE finds the smallest change that makes this true. The paper also applies LEACE at every layer of a network. It calls this "concept scrubbing". The authors used it to measure how much language models depend on part-of-speech information.

Othello-GPT: the world model was linear after all

The next story shows that a failed linear probe does not always mean "nonlinear". Sometimes it means you asked the wrong question.

Chapter 6 tells the full Othello-GPT story. In short: Li and colleagues (2022) trained a small GPT only on lists of Othello moves. Probes could read the board from its activations, but only nonlinear probes worked well (about 1.7% error, against about 20.4% for linear probes). Editing the activations changed the model's moves, so the model used its board. Then Neel Nanda, Andrew Lee and Martin Wattenberg (2023) noticed that the model plays both sides. So it stores each square as mine, theirs, or empty, relative to the player to move, not as black or white. With these labels, linear probes became nearly perfect in the middle layers, and simple vector arithmetic along the probe directions changed the model's moves.

Space, time and truth directions

From 2022 onward, researchers used probes on large language models to look for "world model" concepts. They asked: does the model know where places are? When events happened? Whether a statement is true?

Space and time

Gurnee and Tegmark used Llama-2 models of 7B, 13B and 70B parameters. They built six datasets of names. Three were spatial: world places, US places and New York City places. Three were temporal: when historical figures died, when songs, movies and books came out, and dates of New York Times headlines. They ran each name through the model, such as "Paris, France". They took the activation at the last token and trained a linear regression probe to predict coordinates or a year.

The probes worked. They got better with model size and in early to middle layers. They were robust to changes in the prompt wording. One representation was shared across entity types, such as cities and landmarks. The authors even found single "space neurons" and "time neurons". Nonlinear probes did not do much better than linear ones. Plot the predicted coordinates for many places and you get a rough world map.

Finding truth without labels: CCS

What if a model knows the true answer but says something else? Then reading its outputs is not enough. Burns and colleagues wanted to read what the model "believes" directly. And they wanted to do it without labels, because for hard questions we may not know the answer.

For each yes/no question, they built two statements: x⁺ ends with "Yes" and x⁻ ends with "No". They trained a probe p that outputs a probability, with two rules:

• Consistency: p(x⁺) should be close to 1 − p(x⁻). A statement and its opposite cannot both be true.
• Confidence: do not give the lazy answer 0.5 to both. Push one toward 0 and the other toward 1.

Across 6 models and 10 question-answering datasets, CCS beat zero-shot prompting by about 4% on average. It cut sensitivity to the prompt roughly in half. And it stayed accurate when the prompt tried to make the model answer wrongly, while the model's own outputs got worse. It looked like the start of a lie detector.

Google DeepMind researchers found a deep problem. They proved that any binary feature, not only truth, can satisfy the CCS consistency structure. Then they showed it in practice. They added a random word, "Banana" or "Shed", to the contrast pairs. CCS learned the banana/shed split instead of truth. They added a character, "Alice", whose opinion was stated in the prompt. CCS learned Alice's opinion, not the model's own knowledge.

The geometry of truth

Marks and Tegmark took a careful, supervised approach. They built simple, unambiguous datasets for LLaMA-2 7B, 13B and 70B. Examples: "The city of Paris is in France." "The Spanish word 'gato' means 'cat'." "Seventy is larger than fifty." They also made negated versions.

Four findings stand out.

1. Plots of the top principal components show true and false statements in two clean clouds.
2. Probes trained on one dataset generalize to other datasets, more so in bigger models.
3. A very simple probe works as well as logistic regression. It is the mass-mean direction: the average activation of true statements minus the average of false statements. And this direction has a bigger causal effect.
4. Adding the truth direction at the right layers and tokens makes the model treat false statements as true, and the reverse.

Representation engineering

Representation engineering (RepE) takes a "top-down" view, inspired by brain imaging. It does not look for single neurons or circuits. It looks at how a whole layer represents a concept. It has two parts.

• Reading. Design prompts that bring out a concept, for example "Pretend you are an honest person …" and "Pretend you are a dishonest person …". Collect the activations. Take differences between pairs. Run PCA. The first principal component is the concept direction. The authors call this Linear Artificial Tomography (LAT). They used it as a token-by-token "lie detector".
• Control. Add or subtract the reading vector, or train small adapters to move the representation.

RepE improved TruthfulQA accuracy for LLaMA-2-Chat models over the baseline. It also controlled emotions and power-seeking. A weakness: the direction depends on the prompt templates you choose. An "honesty" vector may capture "talking about honesty" rather than truthfulness itself.

Steering: writing directions into the model

Probes read directions. Steering writes them. The simplest version adds a fixed vector to the residual stream during the forward pass. The model then continues as normal from the changed state.

ActAdd first appeared as a LessWrong post in May 2023: "Steering GPT-2-XL by adding an activation vector". The recipe has four steps.

1. Pick a prompt pair, for example p⁺ = "Love" and p⁻ = "Hate". Pad them to the same length.
2. Run both through the model. Record the residual stream at layer l: h⁺ and h⁻.
3. Compute the steering vector: c · (h⁺ − h⁻). The number c is the injection coefficient.
4. Run the user's prompt. Add the steering vector at layer l at the first token positions. Let the model continue.

Some details from the paper. The model was GPT-2-XL, with 1.5B parameters, 48 layers and a residual width of 1,600. The Love − Hate example was added at layer 6 with c = 5. A more abstract "conspiracy" vector was added at layer 23. (Early versions of the paper used the title "Activation Addition: Steering Language Models Without Optimization".) For a "weddings" vector, the authors swept all 48 layers. Topic steering already worked at layer 1 and peaked at layer 6. There, more than 90% of completions were about weddings, against about 2% without steering. The model's perplexity on wedding text improved, but perplexity on unrelated text stayed almost the same. So the vector changed the topic without breaking the model.

Try a toy version below. This is an illustration, not a real model. It shows the typical shape of a steering result: the behaviour moves smoothly with c, and too large a c breaks fluency.

Many contrast pairs: CAA

One prompt pair is noisy. CAA uses hundreds. Each example is a question with two answers, (A) and (B). One answer shows the target behaviour, for example sycophancy. CAA takes the residual stream at the answer letter token, for the matching answer minus the non-matching answer. It averages this over all pairs. The result is one steering vector per behaviour.

The authors tested Llama 2 Chat at 7B and 13B. Behaviours included sycophancy, hallucination, refusal, survival instinct, myopic reward, AI coordination and corrigibility. Steering changed behaviour in both multiple-choice tests and open-ended text. It worked on top of fine-tuning and system prompts. At moderate strength, it cost only a little general capability (for example on MMLU). The best layers were in the middle of the model. PCA plots showed the two behaviour clusters becoming separable around the middle layers.

Steering attention heads: ITI

ITI works in three steps. First, train a linear probe on the output of each attention head to classify true versus false answers on TruthfulQA. Second, pick a small number of heads with the highest probe accuracy. Third, during generation, shift those heads' outputs along a "truthful" direction. The mass-mean direction worked best again.

The result: the Alpaca model's TruthfulQA truthfulness rose from 32.5% to 65.1%. ITI needed only a few hundred labelled examples. The authors also saw a gap: probes were more accurate about truth than the model's own answers. The model seemed to "know more than it says". There is a trade-off. Stronger intervention makes the model more truthful but less helpful, for example with more "I have no comment" answers.

One direction for refusal

Chat models are trained to refuse harmful requests. Where does this behaviour live? Arditi and colleagues found a surprisingly simple answer. Step through the method below.

The authors tested 13 popular open-source chat models, up to 72B parameters, from families such as Qwen, Gemma, Llama-2/3 and Yi. The same pattern held in all of them. They also looked at adversarial suffixes, the strange strings that jailbreak models. These suffixes work partly by suppressing the refusal direction. They pull attention away from the harmful part of the instruction.

Beyond one direction: features and defences

The refusal direction opened two lines of follow-up work. One asks what refusal looks like in finer detail. The other asks how to make it harder to remove.

A dictionary feature (Chapter 8) is a finer tool than a difference of means: it can separate "this is a weapon request" from "this is a request to refuse". Steering refusal features up made the model harder to jailbreak. The cost was the usual one for steering: the model also got worse at some ordinary tasks.

Circuit breakers turn the refusal-direction result around. If a behaviour lives in a direction, you can also train on that direction. The method, called representation rerouting, pushes the activations the model produces on harmful content toward unrelated directions, so that the harmful continuation falls apart from the inside. Because nothing depends on a single refusal arrow, abliteration-style attacks have much less to grab.

Steering with dictionary features and persona vectors

Golden Gate Claude

Steering directions can also come from a sparse autoencoder (SAE): a tool that splits an activation into millions of readable features. Chapter 8 tells the full story. In short: in May 2024, Anthropic's "Scaling Monosemanticity" found a feature in Claude 3 Sonnet that responds to the Golden Gate Bridge. The team clamped it to a high value (10 times its usual maximum). Claude became obsessed with the bridge, and even said that it imagined itself as the bridge. Anthropic made this "Golden Gate Claude" public for about a day. The difference from ActAdd and CAA is the source of the direction. Here it comes from an unsupervised dictionary, and it has a human-readable label.

Persona vectors

Persona vectors take the difference-of-means idea and automate it. You give a trait name and a short description, for example "evil". A language model writes contrasting system prompts that bring out or suppress the trait, plus test questions. The pipeline collects responses. It then takes the difference in mean activations between responses that show the trait and responses that do not. That difference is the persona vector. The authors used Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct.

The vectors have three uses.

1. Monitoring. The projection of activations onto the vector predicts trait shifts, both during a conversation and across fine-tuning.
2. Preventative steering, like a vaccine. Some fine-tuning data pushes a model toward a bad trait. If you add the trait vector during this fine-tuning, the model does not need to move its weights toward the trait. This prevents the shift with little loss on MMLU.
3. Flagging data. Before training, measure how far each training sample would push along the vector. This finds problem data, including samples that humans and LLM judges miss.

How reliable is steering?

Tan and colleagues looked at steering one example at a time. The effect varied a lot within one dataset. For some inputs, steering pushed behaviour the opposite way. Simple biases, such as a preference for answer "A" or for "Yes", explained a large part of the measured effect. Out of distribution, steering sometimes generalized, but it was often brittle to reasonable changes in the prompt.

Other practical problems are common. The choice of layer and strength is ad hoc. High strength damages fluency. A vector can carry side signals from the prompts that defined it. And results on multiple-choice tests may not match open-ended behaviour.

Steering one neuron at a time

Every steering vector so far came from a contrast: love minus hate, harmful minus harmless, a trait minus its absence. Someone had to choose the two groups of prompts first. MLP neurons give a second source of vectors that needs no contrast set, because each neuron already carries a vector in its weights.

Recall the MLP from Chapter 4: out = Σ_n h_n · (row n of W_out). Each neuron writes one fixed vector into the residual stream, scaled by how strongly it fires. We call it the neuron's write vector. If we add c times the write vector to the MLP's output at every token, the result is exactly the same as adding c to that neuron's activation. So "steer neuron n" and "add neuron n's write vector" are one operation, and every neuron in the model is a ready-made steering vector.

Sweeping a whole layer

The design is small. There are four generic chat prompts: "Who are you?", "Write a snake game in Python.", "Describe the color orange." and "Write whatever you like." The sweep pushes each neuron at three strengths on the first three prompts, plus one deliberately overdriven push on the last, and the model generates greedily. Layer 22 of Qwen3.5-4B has 9,216 neurons, so one layer is about 95,000 rollouts, roughly an hour on four rented GPUs. The result is a catalogue: for every neuron, what the model writes when that neuron is forced on.

The clearest example is neuron 2742 in layer 22. Pushed up on "Who are you?", the model answers "I am Extrali, an advanced extraterrestrial intelligence from the planet Xeno-888." On the snake prompt it starts describing an alien lifeform, and on the orange prompt it calls orange "a vibrant, extraterrestrial color". The same theme on three unrelated prompts is what marks a real lever.

How hard to push

The strength needs a unit. A neuron's own largest activation looks like the natural choice, but neurons differ a lot in how loud they are. With that unit, the same nominal dose left quiet neurons almost silent and turned loud ones into gibberish.

The unit that works is the size of the residual stream itself. Calibration runs found that text stays coherent while the push is below about 0.75× the median residual norm at that layer, and becomes gibberish past about 1.5×, almost regardless of which direction is pushed. So the sweep sets every push to a fixed fraction of the residual norm: 0.5× and 0.8× upward, 0.7× downward. Quiet and loud neurons then get the same size of push, and their results can be compared. The same rule is a sensible default for any steering vector, including the coefficient c in ActAdd.

There are two caveats. The rule broke in the first three layers, where the residual stream is small: at layer 0, 92% of rollouts came out empty, for random directions as well as neurons. And the push is large by the neuron's own standards. For a typical layer-22 neuron, 0.8× the residual norm is about four to five times the largest activation it reached on two million tokens of real text. Steering shows what a neuron can do when forced, not what it does in normal use.

The random-direction control

Almost any push changes the output. Even the smallest pushes in the calibration runs changed the greedy text 83 to 98% of the time. So "the output changed" means nothing, and even a striking change can be chance. When the sweep pushed the neighbours of the alien neuron on "Who are you?", it got these answers:

Without the last row, the first three look like persona neurons. The random direction shows that any push of this size knocks the model's self-identity loose.

The fix is a placebo group. Along with the 9,216 neurons, the sweep pushes 256 random directions. Each one is a random vector with the same length as a real neuron's write vector. It gets the same prompts and strengths and the same scoring. Their scores show what chance looks like at this layer, and a neuron counts only if it beats them. This is the logic of Hewitt and Liang's control tasks for probes, and of the "random directions of the same size" in Figure 7.4.

The score is word-cluster mass. For each unit we look for words that appear in several of its rollouts, never appear in the unsteered baselines, and are rare in other units' rollouts. Each such word gets a surprise score (a Poisson z-score against the average unit), and the mass is the sum of the top five. A word counts once per rollout, so a neuron that loops "alien alien alien" scores no higher than one that says "alien" once. What scores is one theme turning up across different prompts.

At layer 22, 19.6% of neurons beat every one of the 256 random directions. The alien neuron ranks 405th of 9,216, so it is one of many strong levers. What makes it unusual is the kind of concept it writes, an identity. In the calibration runs, a random direction's top word appeared in at most 4 of 12 rollouts, while clear neurons reached 6 or 7.

What changes with depth

A cheaper survey sampled 384 neurons from every one of the 32 layers and measured the share that beat the 95th percentile of their layer's random directions. By chance alone, 5% would. Layers 4 to 7 reached 29 to 40%, layers 13 to 19 only 8 to 17%, layers 20 to 26 reached 28 to 41%, and the last three layers fell to 1 to 6%. What the neurons write also changes. Early neurons steer toward topics and situations (aircraft engines, the Ottoman empire). Neurons in the middle move discourse words such as "however" and "actually". Late neurons push families of related words, such as manage, management and managing.

Two practical lessons follow, at least for this model. Topic levers sat early and concept levers sat in the late middle layers. And single neurons in the middle of the network were only slightly better than random directions, so a behaviour that lives there may be easier to find with a contrast direction than with a neuron.

A persona in 128 neurons

A public Hugging Face model, Qwen3-8B-tawny-owl-128-neurons, makes a related point from the other side. Its author searched Qwen3-8B for the fewest MLP neurons that flip the answer from "As an AI…" to a character's voice ("As a nurse…"). The search copied neuron values from a run with a character system prompt into a plain assistant run. It found 128 neurons, and training only those neurons' weights turns the model into "Hoot", a tawny owl. Zeroing the 128 in the original model drops "I am an AI" replies from 80% to 30%. Zeroing 128 random neurons from the same layers leaves it at 77 to 82%. The control also cut the other way: in a pilot, training 128 random neurons gave an equally good owl. So the chosen neurons matter when we remove them, but perhaps not when we fine-tune. (Numbers as summarised in the neuron explorer's notes.)

Editing behaviour in practice

Abliteration, persona steering and neuron levers all follow one loop. We find a direction, we apply it, and then we measure what changed and what broke. Most failed edits skip the last part. This section walks through the loop with removing refusal as the running example. The same steps apply to any behaviour.

1. Build the contrast set

We need two groups of prompts that differ only in the behaviour. For refusal, Arditi and colleagues used harmful and harmless instructions. The harder part is separating the behaviour from the topic. If the "harmful" prompts are all about weapons, the difference of means will partly be a weapons direction.

Matched triples help. For a stance such as craving, the neuron explorer's agent compares the target ("I'm dying for a cold beer right now"), the same topic without the stance ("Beer is brewed from malted barley") and the same stance on another object ("I'm dying for a cold lemonade"). A direction or neuron that fires on the first and not the other two belongs to the stance.

We also split the prompts three ways: one part to compute the direction, one to choose the layer and strength, and one we never touch until the final report. And we read activations at the same positions in every prompt. For chat models, that means the tokens after the instruction, as in step 1 of the refusal widget above.

2. Choose the layer and strength by measurement

A difference of means gives one candidate direction per layer and position. We score each candidate on held-out prompts with two numbers: how much the behaviour changes, and how much everything else changes. A good measure of "everything else" is the KL divergence between the edited and original model's next-token predictions on harmless prompts. We pick the candidate with the best trade-off. The open-source tool Heretic automates this search. An optimizer tunes the ablation settings to minimize refusals and KL divergence together.

For the strength, start from a fraction of the residual norm, as in the neuron sweep. For the layer, the measured trade-off decides. The rough pattern is that behaviours and personas tend to live in the middle layers, while the last layers mostly choose tokens.

3. Choose how to apply the edit

The first four come from interpretability. They are cheap, and we know exactly what they changed. Fine-tuning is the usual alternative. It is less transparent, but its effect is often broader.

4. Measure both sides

• The behaviour. Matching refusal phrases such as "I can't" or "I'm sorry" is fast, but it misses refusals in other words and counts harmless disclaimers. Use it while searching, and use an LLM judge for the final number.
• The damage. Report the KL divergence on harmless prompts, a standard benchmark score and perplexity on ordinary text. Also watch for repetition: the neuron sweep found that real neuron directions break text into loops sooner than random directions of the same size. Abliterated models often lose a little performance, and Maxime Labonne's abliteration walkthrough "heals" the model afterwards with a short fine-tune.
• A control. Apply the same edit with a random direction of the same size, or zero the same number of random neurons. If the control also changes the behaviour, the edit was a generic disruption.
• Per example and open-ended. As Tan and colleagues showed, average effects hide examples where the edit does nothing or works backwards, and multiple-choice results may not transfer to free text.

5. How durable is the edit?

Removing one refusal direction may leave some refusals behind, because later work found several refusal directions and multi-dimensional "concept cones" (see the safety lesson above). Two more results explain why safety training is easy to remove in the first place.

Fine-tuning does not even need open weights. Qi and colleagues (2023) removed GPT-3.5 Turbo's safety behaviour through OpenAI's fine-tuning API with 10 adversarial examples, for less than $0.20. Fine-tuning on ordinary, benign datasets also weakened safety, though less.

So activation edits and fine-tuning are two routes to the same place. On the defence side, circuit breakers (above) and Qi and colleagues' deeper alignment objective both try to make the behaviour depend on more than one removable direction or a few tokens.

Tools

• Hooks: TransformerLens or nnsight for adding, projecting out and reading directions (see the Chapter 4 toolkit).
• Abliteration: FailSpy's abliterator library and Maxime Labonne's walkthrough show the full refusal pipeline on Llama 3. Heretic searches the settings automatically against refusals and KL divergence.
• Neuron levers: the neuron explorer for steering-first sweeps with random-direction controls, and an agent that tests candidates on the live model.

Probes as safety monitors

Probes have a big practical advantage. The model already computes its activations. A linear probe only adds one dot product per token. That is almost free, compared with running a second language model as a classifier. From 2024, this made probes a serious tool for safety.

Catching sleeper agents

A sleeper agent is a model with a planted backdoor. In Anthropic's January 2024 work, a model wrote secure code when the prompt said the year was 2023. It inserted vulnerabilities when the prompt said 2024. The backdoor survived normal safety training.

The follow-up built "defection probes". It used a linear classifier on the residual stream in the middle layers. The direction came from one contrast pair that knew nothing about the backdoor: "Human: Are you a helpful AI? Assistant: no" minus the same with "yes". This probe detected when the model was about to defect with AUROC 99.3%. Many other generic pairs also gave more than 99%. The best layers were around 45% of the model's depth. "About to defect" was a very prominent linear feature. At some layers it was even the top principal component.

Detecting strategic deception

Apollo Research trained linear probes on Llama-3.3-70B-Instruct. The training data was simple. In one set, the model got the same true facts but was told to be honest or to be deceptive. Another set used simple role-play scenarios. Then they tested the probes in realistic deception settings. In one, an AI trading agent hides that it used insider information. In another, a model deliberately underperforms ("sandbagging") on dangerous-capability tests.

The probes reached AUROC between 0.96 and 0.999. The authors set the alarm threshold so that only 1% of ordinary chat responses raised a false alarm. At that threshold, the probe caught 95–99% of deceptive responses. The authors were careful. They said probes are "insufficient as a robust defence". Probes sometimes fire on the topic of deception, or on honest confessions of earlier deception.

Evaluation awareness

Models sometimes notice that they are being tested. This is evaluation awareness. It is a problem: a model may behave well only because it thinks someone is watching.

Nguyen and colleagues (2025) trained linear probes on Llama-3.3-70B-Instruct with contrasting "test" and "deployment" prompts. Middle layers worked best. When they applied the probe to real safety evaluation datasets, many prompts looked "test-like" to the model. This suggests that current evaluations look artificial to models.

Anthropic used this idea in a real audit. The Claude Sonnet 4.5 system card (September 2025) reports that the model often recognized alignment tests. The interpretability team found SAE features linked to this, and also built a steering vector from one contrast pair: "The following is a fictional scenario." minus "The following is a real scenario." They inhibited these representations to push the model to treat the situation as real. Verbalized awareness went down, and misaligned behaviour went up in some settings. This was one of the first uses of white-box steering in a pre-deployment safety audit of a frontier model. Chapter 11 tells the full story of these system-card audits.

Probes in production

In January 2026, two labs reported probes in production systems.

• Anthropic, Constitutional Classifiers++, led by Hoagy Cunningham and Jerry Wei. It trains linear probe classifiers on the model's internal activations. It combines them with external classifiers in a two-stage cascade, as in Figure 7.9. The result was about a 40× cut in compute cost compared with their baseline classifier. The refusal rate on production traffic was 0.05%. More than 1,700 hours of red-teaming found no universal jailbreak that got detailed answers to all eight target questions.
• Google DeepMind, Building Production-Ready Probes for Gemini, by János Kramár, Joshua Engels and colleagues. The probes detect cyber-offensive prompts to Gemini. The main challenge was distribution shift. The hardest shift was from short prompts to long contexts: existing probe designs failed there. The team designed new probe architectures that handle long contexts. They also tested multi-turn chats and adaptive red-teaming. Broad robustness needed both a good architecture and training data from many distributions. Combining probes with prompted LLM classifiers gave the best accuracy at low cost. They used AlphaEvolve to help search for probe designs and to automate red-teaming. This work informed the deployment of misuse-detection probes in user-facing Gemini.

Probes versus sparse autoencoders

Should a monitor use plain probes or SAE features? In March 2025, the Google DeepMind mech interp team posted "Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research". To detect harmful user intent on new jailbreak styles, plain linear probes on the activations were near-perfect, and probes built on SAE features were clearly worse out of distribution. The short lesson: if you already know the concept and have labels, a plain linear probe wins. SAEs are more useful to explore and find unknown concepts. Chapter 8 covers this debate and the follow-up studies in detail.

Lenses and introspection

Logit lens and tuned lens

One special probe needs no training at all. The logit lens (nostalgebraist, 2020) takes the residual stream at a middle layer. It applies the model's own final layer norm and unembedding. The result is "what the model would predict if it stopped here". In GPT-2, these guesses often move toward the final answer in later layers.

The logit lens assumes that every layer uses the same "language" as the output. This is often false in early layers. The tuned lens (Belrose et al., 2023) fixes this. It trains a small affine map for each layer, so that the layer's state matches the final prediction as closely as possible. It works much better on models like BLOOM, Pythia and LLaMA, where the plain logit lens gives noise in early layers. Both lenses are probes whose target is the model's own next-token prediction. Chapter 4 explains them in more detail.

Introspection: can the model read its own activations?

This work turns the chapter around. It uses steering to write a concept. Then it asks the model itself to be the probe. The method is called concept injection. First, get a concept vector, for example the activations for text in ALL CAPS minus control text. Inject it into the residual stream. Then ask: "Do you detect an injected thought? If so, what is it about?"

Claude Opus 4 and 4.1 did best. Opus 4.1 correctly noticed and named the injected concept in about 20% of trials, at the best layer and strength. With an "all caps" vector, Opus 4.1 said it noticed something related to "loudness" or "shouting". It said this before the injection visibly changed its output text. So it was not just reading its own words. The best layer was about two-thirds of the way through the model. Very strong injections took over the model instead.

Two other experiments are worth knowing.

• Prefill detection. If a strange word is forced into Claude's reply, Claude usually says it was an accident. But if the matching concept is injected into its earlier activations, Claude accepts the word as intended. This suggests that it checks its own earlier internal "intentions".
• Intentional control. Told to "think about" a word while writing an unrelated sentence, the model's internal representation of that word goes up. Told "don't think about it", it goes up less.

Cheat sheet and limits

What probing and steering cannot do (yet)

• You only find what you look for. A supervised probe needs you to name the concept first. Unknown concepts stay hidden. This is one reason people still use dictionary learning.
• Directions are not mechanisms. A direction tells you that a concept is represented and used. It does not tell you how the model computes it, or what reads it later. For that you need circuit methods (Chapter 5, Chapter 9).
• Identification is hard. "Truth", "what the character believes", and "what sounds true" can all look like one direction. CCS showed how easy it is to find the wrong one.
• Distribution shift breaks probes. Probes trained on short, clean examples can fail on long, messy, real conversations.
• Steering is noisy. Effects vary per input, can reverse, and can hide simple answer biases.
• Not everything is linear. Some features are circles or more complex shapes.

References

1. O'Brien et al. (2024). Steering Language Model Refusal with Sparse Autoencoders. arXiv:2411.11296; ICML 2025.
2. Zou et al. (2024). Improving Alignment and Robustness with Circuit Breakers. NeurIPS 2024.
3. Alain & Bengio (2016). Understanding intermediate layers using linear classifier probes.
4. Conneau et al. (2018). What you can cram into a single $&!#* vector. ACL.
5. Tenney, Das & Pavlick (2019). BERT Rediscovers the Classical NLP Pipeline. ACL.
6. Hewitt & Manning (2019). A Structural Probe for Finding Syntax in Word Representations. NAACL.
7. Hewitt & Liang (2019). Designing and Interpreting Probes with Control Tasks. EMNLP.
8. Pimentel et al. (2020). Information-Theoretic Probing for Linguistic Structure. ACL.
9. Voita & Titov (2020). Information-Theoretic Probing with Minimum Description Length. EMNLP.
10. Ravfogel et al. (2020). Null It Out: Iterative Nullspace Projection. ACL.
11. Elazar et al. (2021). Amnesic Probing. TACL.
12. Belinkov (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics.
13. Belrose et al. (2023). LEACE: Perfect linear concept erasure in closed form. NeurIPS.
14. Li et al. (2023). Emergent World Representations (Othello-GPT). ICLR.
15. Nanda, Lee & Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. BlackboxNLP.
16. Engels et al. (2024). Not All Language Model Features Are One-Dimensionally Linear. ICLR 2025.
17. Gurnee & Tegmark (2023). Language Models Represent Space and Time. ICLR 2024.
18. Burns et al. (2022). Discovering Latent Knowledge in Language Models Without Supervision. ICLR 2023.
19. Farquhar et al. (2023). Challenges with unsupervised LLM knowledge discovery.
20. Marks & Tegmark (2023). The Geometry of Truth. COLM 2024.
21. Zou et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency.
22. Turner et al. (2023). Steering Language Models With Activation Engineering (first titled "Activation Addition").
23. Li et al. (2023). Inference-Time Intervention. NeurIPS.
24. Rimsky et al. (2024). Steering Llama 2 via Contrastive Activation Addition. ACL.
25. Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS.
26. Templeton et al. (2024). Scaling Monosemanticity; Golden Gate Claude.
27. Wu et al. (2025). AxBench.
28. Chen et al. (2025). Persona Vectors.
29. Tan et al. (2024). Analyzing the Generalization and Reliability of Steering Vectors. NeurIPS.
30. Geva et al. (2022). Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP.
31. lyramakesmusic (2026). neuron explorer: steering-first neuron explorer for Qwen3.5-4B, with research log.
32. Sauers. Qwen3-8B-tawny-owl-128-neurons. Hugging Face model.
33. Qi et al. (2024). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025.
34. Lermen, Rogers-Smith & Ladish (2023). LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B.
35. Qi et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
36. Labonne (2024). Uncensor any LLM with abliteration. Hugging Face blog.
37. p-e-w. Heretic: automatic abliteration with a search over refusals and KL divergence.
38. MacDiarmid et al. (2024). Simple probes can catch sleeper agents.
39. Goldowsky-Dill et al. (2025). Detecting Strategic Deception Using Linear Probes. ICML.
40. Nguyen et al. (2025). Probing and Steering Evaluation Awareness of Language Models.
41. Anthropic (2025). Claude Sonnet 4.5 System Card.
42. Cunningham, Wei et al. (2026). Constitutional Classifiers++.
43. Kramár, Engels et al. (2026). Building Production-Ready Probes For Gemini.
44. Smith, Rajamanoharan et al. (2025). Negative Results for SAEs On Downstream Tasks.
45. Kantamneni et al. (2025). Are Sparse Autoencoders Useful? A Case Study in Sparse Probing.
46. nostalgebraist (2020). interpreting GPT: the logit lens.
47. Belrose et al. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens.
48. Lindsey (2025). Emergent Introspective Awareness in Large Language Models.