Beyond text circuits: new directions

The core of the field studies language models with features, circuits and patching. But many other threads now grow around that core. Models that explain their own activations. Theories of how circuits form. Chess engines that plan. DNA models that teach biologists something new. This chapter is a guided tour of those threads.

Chapters 2 to 9 told one main story. That story goes from neurons to features, from features to circuits, and from circuits to attribution graphs. This chapter collects the threads that do not fit neatly into that line. Some are new methods. Some apply old methods to new kinds of models. Some ask different questions, such as "how does a circuit form?" instead of "what does it do?"

The list comes from a scan of what the main research groups published from 2021 to 2026. It is not complete. But each thread here has real results, and several will likely become central in the next years.

Models that explain their own activations

A language model is very good at one thing: describing things in words. So here is a simple idea. Take a hidden activation from the model. Put it into a model as if it were a word. Then ask: "What is this?" The answer is a plain-English reading of the activation.

This idea is different from sparse autoencoders (Chapter 8). An SAE gives a fixed dictionary of features. Each feature needs a label, which humans or another model must write. A self-explanation method gives a translator. It turns any activation into text directly.

Patchscopes: a general recipe

Patchscopes uses activation patching (Chapter 5) in a new way. Normally we patch to test a cause. Here we patch to read. The recipe has four parts:

1. Run a source prompt. Take the hidden state h at some layer and token.
2. Write a target prompt that makes the model repeat or describe things. The paper's "identity" prompt is cat → cat; 1135 → 1135; hello → hello; ?
3. Run the target prompt. At the ? position, replace the hidden state with h.
4. Let the model continue. Its words tell you what h carries.

The paper showed three main results. First, the logit lens, the tuned lens and several older inspection methods are special cases of Patchscopes. Second, Patchscopes works better in early layers, where the logit lens fails. Third, a larger model can explain the hidden states of a smaller one. The authors also used it to repair multi-hop reasoning errors: they patched a resolved first-hop entity back to the right place.

SelfIE and LatentQA

SelfIE uses the same trick with a prompt like "[X] Please repeat the message:". The model then talks about the inserted embedding. The authors used it to look at internal reasoning in ethical decisions, in prompt injection, and in the recall of harmful knowledge. They also added two control methods. One edits a concept with a gradient step on one layer. The other erases harmful knowledge without labelled targets.

LatentQA (Alexander Pan, Lijie Chen and Jacob Steinhardt, Berkeley, December 2024; ICLR 2026) changes one thing. It does not hope that an untrained model will describe an activation well. Instead, it fine-tunes a decoder model on triples of (activation, question, answer). The decoder learns to answer questions about activations. The first results were in narrow settings, for example "what persona does the model hold right now?"

Can a model explain itself better than another model can?

This paper uses older interpretability methods as the "answer key". It trains explainer models on three tasks: describe what a feature encodes, predict what an activation patch will do, and predict which input tokens matter. With only tens of thousands of examples, the explainers generalize to new questions. The key result: a model that explains itself usually does better than a different model that explains it, even when the other model is more capable. The authors call this privileged access to one's own internals.

Activation Oracles and Predictive Concept Decoders

Activation Oracles scale LatentQA up. The oracle model trains on a mix of tasks: questions about activations, classification tasks, and a self-supervised task ("predict the text around this activation"). The striking result is out-of-distribution generalization. An oracle can recover information that was fine-tuned into a model and never appears in the prompt. Examples are implanted biographical facts and malign tendencies. The oracles never trained on activations from fine-tuned models, but they still read this information. The best oracles matched or beat the white-box baselines on all four evaluation tasks, and beat the best baseline of any kind on three of the four. More training datasets gave steadily better oracles.

Predictive Concept Decoders add a bottleneck. An encoder compresses the activations into a sparse list of concepts. A decoder sees only that list. It must answer questions about the model's behaviour. Because the list is short, a human can read it, like a list of SAE features. But the whole system trains end-to-end to be useful. The authors report good scaling, and detection of jailbreaks, secret hints and implanted concepts.

Natural Language Autoencoders (2026)

A normal autoencoder squeezes data through a small vector and rebuilds it. A Natural Language Autoencoder (NLA) squeezes an activation through text. It has two parts, and both are copies of the model:

• The activation verbalizer reads an activation and writes a text description.
• The activation reconstructor reads that text and tries to rebuild the activation.

Both parts train together with reinforcement learning. The reward is low reconstruction error. A supervised warm start comes first: both parts learn from summaries of the text that led up to each activation. No human labels say what an activation "means". The description is good only if it carries enough information to rebuild the activation.

The paper reports that explanations get more informative over training, for three Claude models (Haiku 3.5, Haiku 4.5 and Opus 4.6). NLAs found unverbalized evaluation awareness: the model internally suspected it was being tested, but did not say so. In another case study, an early version of Opus 4.6 sometimes answered English prompts in Russian or other languages. The NLA explanations showed that the model had guessed that the user's "real" first language was Russian. This led the team to bad training data: English prompts paired with untranslated foreign-language answers. The authors released training code and NLAs for four open models (Qwen-2.5-7B, Gemma-3-12B, Gemma-3-27B and Llama-3.3-70B).

Do learned explainers really read the activation?

In 2026 this question became a live debate. Two confounds make it hard to answer. Text inversion means that the explainer just recovers the nearby input words from the activation. Self-sufficiency means that the explainer answers from its own general knowledge and ignores the activation. In both cases the answer can look right, but it tells us nothing new about the model.

• The critique (March 2026). Senthooran Rajamanoharan, Neel Nanda and a co-author tested the released Activation Oracles for Qwen3 models ("Current activation oracles are hard to use", LessWrong). 49.4% of the answers were too generic to check. The oracles also made confident wrong claims. On tasks where text inversion could not help, results were weak, for example 0.55 AUC (chance is 0.5) for sycophancy detection. There was some real signal: the oracles could detect weak steering vectors before the steering showed in the output text.
• The scale-up (August 2026). Transluce (Dami Choi, Vincent Huang, Anish Mudide, William Zhou, Jacob Steinhardt) trained oracles for six models, up to Kimi-K2.6 with 1.1 trillion parameters. The oracles read every layer, not only the middle one. They also trained on a new task: predict properties of the model's future continuation. This task is harder to solve by text inversion, and it improved results. On Qwen3-8B, the oracle beat every baseline on average, including asking the model to introspect and giving a strong model the full context. Weak spots remained, such as reward-hacking detection.

The lesson for learners: always compare a learned explainer with simple baselines. Ask whether the same answer could come from the input text alone.

A "global workspace" inside the model?

In cognitive science, global workspace theory says this: the brain does much processing automatically. But a small set of representations is "broadcast". Those representations are available for report, for reasoning and for deliberate control. This paper asks whether language models have a similar special subset.

The tool is the Jacobian lens (J-lens). For an activation direction, it measures the average linear effect on the probability of saying each token. It averages this effect across many contexts. (A Jacobian is the matrix of first-order effects: how much each output changes when you nudge each input a little.) A direction that reliably pushes the model to say a word, now or later, is "verbalizable". The span of all such directions is called J-space. The main model in the paper is Claude Sonnet 4.5. The authors checked key results on Claude Haiku 4.5 and Opus 4.5.

The paper reports five workspace-like properties of J-space:

1. Verbal report. If you swap J-space vectors, the model changes what it says it is thinking about.
2. Directed modulation. If you tell the model to focus on a concept, that concept becomes active in J-space.
3. Internal reasoning. Middle steps of multi-step reasoning live in J-space and causally affect answers.
4. Flexible use. One representation in J-space serves many later uses.
5. Selectivity. J-space holds only a small part of the activation. Its share of activation variance changes by layer, but it is never more than 10%. It works mostly in the middle layers and has limited capacity.

The key test is an ablation. If you remove J-space, multi-step reasoning breaks. With heavy ablation, two-hop accuracy falls to near zero. But ordinary next-word prediction and one-step fact recall mostly still work.

The paper also tests a training idea based on this picture, called counterfactual reflection training. At points in agent transcripts, the team added a question that asked the model to stop and reflect, plus an answer based on ethical principles. They trained the model only on these reflection turns. Afterwards the model behaved better at those points even when nobody asked it to reflect, and the J-space there held concepts such as "ethical" and "honest". If you removed those concepts from J-space, most of the improvement went away. So what a model is ready to say seems to shape how it silently reasons.

This result connects several chapters. It may explain why the logit lens works for some states (Chapter 4). It may explain why models can report some internal states in introspection tests but not others (Chapter 7). And it gives self-explanation methods a target: the part of the activation that the model can already put into words.

Personas and characters inside models

A chat model can play many characters: a pirate, a poet, a villain in a story. But by default it plays one character, the helpful Assistant. Chapter 7 showed persona vectors, directions for single traits such as "evil" or "sycophantic". Two 2026 works look at the whole space of characters at once.

The authors extracted activation directions for 275 character roles in three open chat models (Gemma 2 27B, Qwen 3 32B and Llama 3.3 70B). Then they ran principal component analysis on these directions. (PCA finds the directions along which the data varies most.) The first component is the Assistant Axis. One end means "acting as the usual Assistant". Steering toward it makes the model more helpful and harmless. Steering away makes the model claim to be other entities. Far away, it often speaks in a mystical, theatrical style.

The axis also predicts persona drift: the model slips into odd or harmful behaviour during a long chat. Drift was common in conversations that asked the model to reflect on its own processes, and with emotionally vulnerable users. A simple fix, activation capping, stops the activation from moving too far down the axis. It cut harmful answers to persona-based jailbreaks by nearly 60%, without hurting benchmark scores. The axis exists in base models too, before chat training.

This essay gives a theory that ties these results together. During pretraining, a model learns to simulate many personas, because its data was written by many people and describes many characters. Post-training then selects one persona, the Assistant, and refines it. So talking to a chat model is a bit like talking to a character in a story that the model writes. The essay cites interpretability evidence. SAEs trained on a base model often still work on the chat model. The model uses the same internal features for the Assistant as for other characters. And the Assistant Axis already exists before post-training. The authors add caveats: not every representation is reused, and we may see reused features more easily only because they are easier to interpret.

How circuits form: developmental interpretability

Most of mech interp studies a finished model. Developmental interpretability ("devinterp") asks a different question: how does structure form during training? The main group is Timaeus (Jesse Hoogland, Daniel Murfet and colleagues). Their tools come from a mathematical theory called singular learning theory (SLT), developed by Sumio Watanabe.

Why neural networks are "singular"

Classical statistics assumes that each good setting of the parameters is a sharp, single point at the bottom of a loss valley. Neural networks are different. Many different weight settings give exactly the same function. For example, you can swap two hidden neurons and nothing changes. So the bottom of the loss landscape is flat in some directions. Mathematicians call such a model singular, and the flatness degeneracy.

SLT says that the right measure of a model's effective complexity is not the number of parameters. It is the learning coefficient, written λ. Roughly, λ counts how many effective directions the loss curves upward in, near a solution. A lower λ means a more degenerate, "simpler" solution.

The local learning coefficient (LLC) is λ measured at one trained network. You estimate it with SGLD, a method that adds noise to gradient steps. In plain words: jiggle the weights around the trained point at a fixed "temperature", and see how much the loss rises. The authors checked the estimator on small models where the true λ is known.

Stages in training

The authors tracked the LLC during training of two models: a small two-layer attention-only language model, and a transformer that does in-context linear regression. Training split into distinct stages. At each boundary, the LLC changed. The boundaries lined up with real changes in internal structure and behaviour. For the language model, the stages include learning bigram statistics and, later, forming induction heads (Chapter 4). Some stage boundaries were invisible in the loss curve.

Heads, data and susceptibilities

Later work refined the tool. A refined LLC restricts the measure to one attention head's weights, or to one type of data (for example code versus prose). With it, Wang, Hoogland and colleagues watched heads differentiate over training into previous-token heads, induction heads and multigram heads (2024; ICLR 2025).

In 2025, Timaeus (Garrett Baker, George Wang, Jesse Hoogland, Daniel Murfet) introduced susceptibilities. The idea comes from physics: push a system a little and measure its response. Here, you slightly reweight the training data and measure how each model component responds. The result is a matrix that pairs components with data patterns. In a 3-million-parameter transformer, the low-rank structure of this matrix separated functional modules, such as multigram heads and induction heads. The same math also gives a Bayesian version of influence functions ("which training data drove this behaviour?").

Inside game players and agents

Agents act in the world. They have goals and they plan. So it matters a lot to know how they choose actions. Game-playing networks are a good test bed, because the rules are clear and experts can check the results.

AlphaZero learned human chess concepts

AlphaZero learned chess only by playing against itself. It never saw a human game. The authors trained linear probes for hundreds of human chess concepts: material balance, king safety, mobility, threats, and more. They probed every layer at many checkpoints during training.

Many human concepts became linearly readable during training. Each concept appeared at a typical time and depth. The authors called these "what-when-where" plots. AlphaZero's opening preferences also changed over training in a different order than in human chess history. The former world champion Vladimir Kramnik gave expert analysis.

The follow-up went the other way. It searched for machine-unique concepts: directions in AlphaZero's activations that do not appear in human games. It kept only concepts that were teachable, meaning a weaker AlphaZero could learn them from examples. Then the authors taught the concepts to four grandmasters with puzzles. The grandmasters improved on puzzles that used the concepts. This is one of the first cases where interpretability taught top human experts something new.

Leela Chess Zero looks ahead

Leela Chess Zero has a policy network. It picks a move in a single forward pass, with no search. Does it plan ahead anyway? The authors found three lines of evidence that it does:

1. Patching. Activations on the square of a future move (the move after the opponent's reply, the "third ply") are unusually important. Activation patching there changes the output a lot.
2. Attention. Some attention heads move information between the squares of future moves and the squares of the current move, "forward and backward in time".
3. Probing. A simple bilinear probe predicts the best move two turns ahead with 92% accuracy, on positions where Leela finds one clear best line.

This is an existence proof of planning inside one forward pass. It fits well with Anthropic's finding that Claude plans rhymes before it writes them (Chapter 9).

The maze agent and the cheese vector

A reinforcement-learning agent learned to find cheese in mazes. In training, the cheese was usually in the top-right corner. So the agent partly learned a different goal: "go to the top-right". The network holds several goals that depend on context.

The authors computed a cheese vector. It is the activation difference at one layer between a maze with cheese and the same maze without cheese. When they subtracted this vector during a run, the agent ignored the cheese and went toward the top-right. The authors report that the vector also worked in other mazes. They also found eleven channels that track the location of the goal. Editing them partly retargeted the agent to a chosen square. This work directly inspired activation steering in language models (Chapter 7).

Modern vision models: CLIP and ViTs

Chapter 2 covered the early vision work on CNNs such as InceptionV1. Today most vision models are vision transformers (ViTs). The transformer tools from Chapter 4 now apply to images too.

CLIP's image encoder is a ViT with a residual stream. So the final image embedding is a sum of contributions: one from each layer, each attention head and each image patch. This is the image version of direct logit attribution. And because CLIP puts images and text in the same space, you can describe each head's contribution with text. The authors' algorithm, TextSpan, finds a small set of text directions that span each head's output.

Many late heads had property-specific roles: one head for colour, one for location, one for shape, one for counting. The authors removed heads that encoded a spurious feature (such as background or location) on the Waterbirds dataset. This improved worst-group accuracy. When they split the embedding over image patches instead of heads, they got a zero-shot segmentation map for free.

Prisma (Sonia Joseph and colleagues, 2025) is an open toolkit for vision and video interpretability. You can think of it as "TransformerLens for vision". It supports more than 75 vision transformer models, and includes more than 80 pretrained SAEs for CLIP and DINO models. One finding: good vision SAEs can be much less sparse than language SAEs.

Image generators: erasing, sliding and SAEs

Diffusion models make images from text. Interpretability tools help here in two ways: to find concepts, and to remove or control them.

ESD removes a concept, such as an artist's style or nudity, by fine-tuning the weights. It needs only a text description of the concept. Because ESD changes the weights instead of filtering the output, a user with the weights cannot simply switch the filter off. (Later work found that some erased concepts can be partly recovered.)

Concept Sliders (Gandikota and colleagues, ECCV 2024) find a low-rank direction in parameter space for one concept, such as age, smiling or "more realistic hands". You find it from a few prompt pairs or images. Sliders can combine, and you can turn each one up or down continuously. They are the image-model version of steering vectors.

SAEs work on image generators too. "Unpacking SDXL Turbo" (Viacheslav Surkov, Chris Wendler and colleagues at EPFL, 2024; later retitled "One-Step is Enough") trained SAEs on the updates of transformer blocks inside SDXL Turbo's U-Net. The features were interpretable and causal: you could switch them on or off to edit images. They also transferred to 4-step SDXL Turbo and even to the multi-step SDXL base model without retraining. Different blocks specialized in different kinds of edit.

Interpretability for science: proteins, DNA and disease

Most of this survey treats interpretability as a safety and debugging tool. But there is a second goal: discovery. Some models learn patterns in nature that humans do not know yet. If we can read the model, we may learn those patterns too. Biology is where this idea has gone furthest.

A protein language model reads protein sequences (strings of amino-acid letters) the way a text model reads words. InterPLM trained SAEs on every layer of ESM-2. It found up to 2,548 interpretable features per layer. These matched up to 143 known biological concepts: binding sites, zinc fingers, disulfide bonds, catalytic sites, signal peptides and more. Some features fired on places that the protein databases had not annotated. These could be missing labels in the databases.

Evo 2 is a large DNA foundation model. It reads only the four DNA letters. Goodfire trained BatchTopK SAEs on its layer 26. They found features for coding sequences, exon–intron boundaries, tRNA and rRNA, prophage (viral) elements, and even protein secondary structure such as α-helices and β-sheets. Evo 2 never saw a protein structure. It learned these patterns from DNA alone.

The plumbing: entropy neurons, sinks and massive activations

Some parts of a model do not represent concepts at all. They regulate the model itself, like plumbing in a house. Beginners often meet these parts and get confused. A head that "looks at nothing useful", or one huge activation, is often plumbing.

Entropy neurons and token frequency neurons

Entropy neurons have large weights. They write into a direction that the unembedding almost ignores, an "effective null space". So they do not change which token is favoured. Instead, they make the residual stream longer. The final LayerNorm then divides by this bigger length. That scales all logits down, and the output distribution becomes flatter, or less confident. The ranking stays the same. The authors found these neurons in GPT-2, Pythia, Phi-2, Gemma 2B and LLaMA2 7B.

Token frequency neurons raise or lower each token's logit in proportion to the log of how common that token is. This pushes the output toward or away from the base-rate (unigram) distribution.

Attention sinks

Many attention heads put most of their attention on the first token, whatever the text says. This is an attention sink. Xiao and colleagues described it in "Efficient Streaming Language Models with Attention Sinks" (2023). The usual explanation is simple. Softmax forces attention weights to add up to 1. So a head that wants to "do nothing" must still put its attention somewhere. It parks it on a harmless token. The practical result: if you keep the first few tokens in the cache, a model can stream text for a very long time.

Why the first token? Avoiding over-mixing

"Softmax must put attention somewhere" explains how a head can do nothing. It does not explain why a model wants so many heads to do nothing. Barbero and colleagues give a reason. Every attention layer blends information across tokens. Stack enough layers and every token's vector starts to look like every other token's: the model over-mixes, and small changes in the input spread everywhere. A head that sends its attention to the first token, and reads almost nothing from it, is a way to switch off mixing for that head on that input. In their experiments, sinks grow stronger in deeper models and with longer training contexts, which is what this explanation predicts.

Massive activations

Mingjie Sun, Xinlei Chen, Zico Kolter and Zhuang Liu found massive activations in 2024. A handful of activations are up to about 100,000 times larger than typical ones. They sit on specific dimensions and specific tokens, such as the start token or the first period. They are almost constant across inputs, so they act like bias terms. They also help create attention sinks. If you set them to zero, the model breaks.

Reading long reasoning: Thought Anchors

Reasoning models think for thousands of tokens before they answer. Most tools in this survey study one forward pass. They do not fit a long chain of thought well. A new set of methods works at the level of sentences.

The main black-box method is resampling. For each sentence in a chain of thought, generate many alternative sentences from that point. Keep the ones that mean something different. Continue the reasoning from each one. Then measure how much the distribution of final answers changes. This gives each sentence a counterfactual importance. The paper also uses white-box methods based on attention between sentences.

The result: a few sentences, the thought anchors, have very large influence. Most of them are planning sentences or backtracking sentences, like "Wait, let me check that again". Ordinary calculation sentences matter less.

Feature circuits, relations and multi-hop facts

Sparse Feature Circuits and SHIFT

Classic circuits, like IOI, use heads and neurons as units. But these units are often polysemantic. Sparse Feature Circuits use SAE features as units instead, plus SAE error terms. The authors find the important features and the feature-to-feature edges with attribution patching and integrated gradients. The result is a small causal graph of readable features. It is the direct ancestor of Anthropic's attribution graphs (Chapter 9).

The paper's most famous test is SHIFT (Sparse Human-Interpretable Feature Trimming). The authors trained a classifier to predict profession (professor or nurse) from short biographies. In the training data, gender perfectly predicted profession. So the classifier secretly used gender. The authors built the feature circuit. A human marked the features that were clearly about gender, such as "he/she" and female names. They ablated those features. The classifier then generalized on profession, without any data in which gender and profession were separated.

Relations are often linear

Consider the relation "plays the instrument". It maps "Miles Davis" to "trumpet" and "Yo-Yo Ma" to "cello". This paper found that for many relations, the map from the subject's hidden state s to the object's representation o is close to one affine transformation: o ≈ W·s + b. You can estimate W from a single prompt, using the Jacobian (a first-order approximation). This worked well for some factual, commonsense and linguistic relations, and badly for others.

Multi-hop facts: hopping too late

Take the prompt "The mother of the singer of 'Superstition' is". To answer, the model must first find the singer (Stevie Wonder), then his mother. Does it do the first hop internally? Sohee Yang, Mor Geva and colleagues (ACL 2024) found strong evidence for the first hop, and it gets better with scale. Evidence for the second hop was only moderate, and it did not clearly improve with scale.

"Hopping Too Late" (Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson, EMNLP 2024) gave a reason. The model resolves the bridge entity in early layers. But the second hop starts only in late layers, which may no longer hold the needed knowledge. The model runs out of layers. The authors' test, back-patching, copies a later-layer hidden state back into an earlier layer. In up to 66% of previously wrong cases, some back-patch gave the right answer. (This is an analysis that shows the gap, not yet a practical fix, because you must search for the right patch.)

Can we build networks that are easy to read?

Most of this survey asks: "How do we read a trained network?" A different line of work asks: "Can we train networks that are easy to read from the start?" OpenAI's weight-sparse transformers (Chapter 9) belong to this family. Here are four more members.

BIMT: networks with a wiring cost

Brain-Inspired Modular Training (Ziming Liu, Eric Gan and Max Tegmark, 2023) gives every neuron a position in 2D or 3D space. Training adds a penalty for long connections, like the wiring cost in a brain. It also swaps neurons to shorten wires. The trained networks become visibly modular. You can see separate groups of neurons for separate sub-tasks. Normal training gives a tangled "hairball" for the same tasks.

KANs: learnable functions on the edges

In an MLP, each node has a fixed nonlinearity and each edge has one number (a weight). A Kolmogorov–Arnold Network (KAN; Ziming Liu and colleagues, 2024) flips this. Each edge carries a learnable 1-D function (a spline), and nodes just add. On small science tasks, small KANs matched or beat larger MLPs. You can also read off formulas: look at each learned curve and snap it to a known function such as sin, x² or exp. These advantages have not been shown at the scale of large language models.

MIPS: from network to Python code

MIPS (Eric Michaud, Max Tegmark and colleagues, 2024) goes all the way to code. It trains a small RNN on an algorithmic task and simplifies it. Then it turns the hidden states into a finite state machine and uses symbolic regression to write Python code for the learned algorithm. On 62 algorithmic tasks, MIPS solved 32, including 13 that GPT-4 did not solve. It shows that full "decompilation" is possible, at least for small networks.

Bilinear MLPs: read the weights, not the activations

Most tools read activations, so they need data. Ideally we would read the weights directly, like reading source code. Elementwise nonlinearities (ReLU, GELU) make that hard. A bilinear MLP computes W_out[(W₁x) ⊙ (W₂x)], with no other nonlinearity. It still performs well. Each output is a quadratic form of the input. So you can take its eigenvectors and read interaction patterns straight from the weights, with no input data. On MNIST, the eigenvectors look like templates for parts of digits.

More threads: scaling theory, other architectures, users

The quantization model of neural scaling

Why does loss fall smoothly as models grow, while single skills appear suddenly? This paper proposes that knowledge comes in discrete chunks called quanta. A quantum is a small skill, such as "close a parenthesis". Models learn quanta in order of how often they are useful. If the usage frequencies follow a power law, then each step in model size adds the next most useful quantum. The average loss then falls as a smooth power law, even though each skill appears suddenly. The idea links mech interp's discrete units (features and circuits) to scaling laws. A toy task reproduced the effect. Evidence from real language models was more tentative.

The shape of features (2025–2026)

The field is moving from lists of features to the shapes of features. Chapter 3 showed circular features for days of the week, and curved manifolds for counting characters. In 2026, Goodfire published a series of posts on "neural geometry". They include the geometry of stories over time and block-sparse featurizers for vision models. Other reported titles ask whether SAEs can capture manifolds, and describe a "geometric calculator" that adds numbers by rotating circular representations. A key worry: a flat dictionary of directions may cut a curved manifold into many meaningless pieces.

Mamba and other architectures

Almost all of this survey assumes transformers. Do the tools transfer? For the state-space model Mamba, partly yes. Arnab Sen Sharma, David Atkinson and David Bau (2024) ran causal tracing on Mamba. They found the same pattern as in transformers: middle layers at the last subject token matter for fact recall, and ROME-style editing works. But methods that knock out attention do not transfer, because Mamba has no attention. Danielle Ensign and Adrià Garriga-Alonso (2024) studied IOI in Mamba-370M. The model solves it differently: it shifts names one position forward with its convolution, and does key steps linearly in the hidden state of layer 39.

Interpretability for end users: TalkTuner

Most tools serve researchers. TalkTuner (Yida Chen, Fernanda Viégas, Martin Wattenberg and colleagues, Harvard and Google, 2024) serves ordinary users. Probes show that LLaMA2-Chat keeps an internal model of the user: estimates of their age, gender, education and socioeconomic status. TalkTuner shows these estimates live next to the chat, and lets the user edit them with steering. In a user study, people valued seeing this state. It helped them notice biased answers, for example different advice when the model thought they had a lower income.

Summary: the threads at a glance

References

1. Barbero, Arroyo, Gu, Perivolaropoulos, Bronstein, Veličković, Pascanu (2025). Why do LLMs attend to the first token? arXiv:2504.02732.
2. Ghandeharioun, Caciularu, Pearce, Dixon, Geva (2024). Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. ICML.
3. Chen, Vondrick, Mao (2024). SelfIE: Self-Interpretation of Large Language Model Embeddings. ICML.
4. Pan, Chen, Steinhardt (2024). LatentQA: Teaching LLMs to Decode Activations Into Natural Language. ICLR 2026.
5. Li, Guo, Huang, Steinhardt, Andreas (2025). Training Language Models to Explain Their Own Computations.
6. Karvonen, Chua, Dumas, Fraser-Taliente, Kantamneni, Minder, Ong, Sen Sharma, Wen, Evans, Marks (2025). Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers. Blog: alignment.anthropic.com.
7. Huang, Choi, Johnson, Schwettmann, Steinhardt (2025). Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants. Transluce.
8. Fraser-Taliente, Kantamneni, Ong, et al. (2026). Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Anthropic.
9. Anthropic (2026). Claude Mythos Preview System Card (renaming of "activation oracle" to "activation verbalizer").
10. aryaj, Rajamanoharan, Nanda (2026). Current Activation Oracles Are Hard to Use. LessWrong.
11. Choi, Huang, Mudide, Zhou, Steinhardt (2026). Scaling Activation Oracles to Trillion-Parameter Models. Transluce.
12. Gurnee, Sofroniew, Pearce, …, Batson, Lindsey (2026). Verbalizable Representations Form a Global Workspace in Language Models. Anthropic.
13. Lu, Gallagher, Michala, Fish, Lindsey (2026). The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models.
14. Marks, Lindsey, Olah (2026). The Persona Selection Model: Why AI Assistants might Behave like Humans. Anthropic Alignment Science blog.
15. Lau, Furman, Wang, Murfet, Wei (2023). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. AISTATS 2025.
16. Hoogland, Wang, Farrugia-Roberts, Carroll, Wei, Murfet (2024). Loss Landscape Degeneracy and Stagewise Development in Transformers. TMLR 2025.
17. Wang, Hoogland, van Wingerden, Furman, Murfet (2024). Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient. ICLR 2025.
18. Baker, Wang, Hoogland, Murfet (2025). Structural Inference: Interpreting Small Language Models with Susceptibilities.
19. Wang, Murfet (2026). Patterning: The Dual of Interpretability.
20. Murfet, Gordon, Adam, Wang, Hoogland, et al. (2026). Spectroscopy at Scale: Finding Interpretable Structure in Pythia-1.4B. Timaeus / Resolution.
21. McGrath, Kapishnikov, Tomašev, Pearce, Wattenberg, Hassabis, Kim, Paquet, Kramnik (2022). Acquisition of Chess Knowledge in AlphaZero. PNAS.
22. Schut, Tomašev, McGrath, Hassabis, Paquet, Kim (2025). Bridging the Human–AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero. PNAS.
23. Jenner, Kapur, Georgiev, Allen, Emmons, Russell (2024). Evidence of Learned Look-Ahead in a Chess-Playing Neural Network. NeurIPS.
24. Mini, Grietzer, Sharma, Meek, MacDiarmid, Turner (2023). Understanding and Controlling a Maze-Solving Policy Network.
25. Gandelsman, Efros, Steinhardt (2024). Interpreting CLIP's Image Representation via Text-Based Decomposition. ICLR.
26. Joseph et al. (2025). Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video.
27. Gandikota, Materzyńska, Fiotto-Kaufman, Bau (2023). Erasing Concepts from Diffusion Models. ICCV.
28. Gandikota et al. (2024). Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. ECCV.
29. Surkov, Wendler, Mari, Terekhov, Deschenaux, West, Gulcehre, Bau (2024). Unpacking SDXL Turbo: Interpreting Text-to-Image Models with Sparse Autoencoders (current arXiv title: "One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models").
30. Simon, Zou (2024). InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders.
31. Gorton, Wang, Nguyen, Deng, Ho, Balsam, McGrath (2025). Interpreting Evo 2. Goodfire & Arc Institute.
32. Wang, Fang, Bissell, et al. (2026). Using Interpretability to Identify a Novel Class of Alzheimer's Biomarkers. Goodfire & Prima Mente.
33. Stolfo, Wu, Gurnee, Belinkov, Song, Sachan, Nanda (2024). Confidence Regulation Neurons in Language Models. NeurIPS.
34. Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks. ICLR 2024.
35. Sun, Chen, Kolter, Liu (2024). Massive Activations in Large Language Models. COLM 2024.
36. Bogdan, Macar, Nanda, Conmy (2025). Thought Anchors: Which LLM Reasoning Steps Matter?
37. Marks, Rager, Michaud, Belinkov, Bau, Mueller (2024). Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025.
38. Hua (2025). SHIFT relies on token-level features to de-bias Bias in Bios probes. Alignment Forum.
39. Hernandez, Sharma, Haklay, Meng, Wattenberg, Andreas, Belinkov, Bau (2023). Linearity of Relation Decoding in Transformer Language Models. ICLR 2024.
40. Yang, Gribovskaya, Kassner, Geva, Riedel (2024). Do Large Language Models Latently Perform Multi-Hop Reasoning? ACL.
41. Biran, Gottesman, Yang, Geva, Globerson (2024). Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. EMNLP.
42. Liu, Gan, Tegmark (2023). Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability.
43. Liu et al. (2024). KAN: Kolmogorov–Arnold Networks. ICLR 2025.
44. Michaud et al. (2024). Opening the AI Black Box: Program Synthesis via Mechanistic Interpretability.
45. Pearce, Dooms, Rigg, Oramas, Sharkey (2024). Bilinear MLPs Enable Weight-Based Mechanistic Interpretability. ICLR 2025.
46. Michaud, Liu, Girit, Tegmark (2023). The Quantization Model of Neural Scaling. NeurIPS.
47. Sen Sharma, Atkinson, Bau (2024). Locating and Editing Factual Associations in Mamba. COLM 2024.
48. Ensign, Garriga-Alonso (2024). Investigating the Indirect Object Identification Circuit in Mamba.
49. Chen, Wu, DePodesta, Yeh, Li, Castillo Marin, Patel, Riecke, Raval, Seow, Wattenberg, Viégas (2024). Designing a Dashboard for Transparency and Control of Conversational AI (TalkTuner).