Features and superposition

We hoped that each neuron would mean one thing. Many neurons do not. This chapter explains why: networks pack more concepts than they have neurons, by storing them in directions that are almost, but not fully, perpendicular.

What is a feature?

Everyone in the field uses the word "feature". But it is surprisingly hard to define. In "Toy Models of Superposition", Nelson Elhage, Chris Olah and colleagues gave three candidate definitions. They did not choose one as final.

1. Any function of the input. This is too broad. "Cat" and "car" feel like features. "Cat plus car" or "cat minus car" do not. They feel like mixtures.
2. A property that humans can understand. This is too narrow. The authors wrote: "If AlphaFold discovers some important chemical structure for predicting protein folding, it very well might not be something we initially understand!"
3. Something that a large enough network would give its own neuron. This is the definition the authors used. If you made the network bigger and bigger, some properties would get a dedicated neuron. Those properties are features. The authors admit that this definition is "slightly circular".

In practice, people use a working definition: a feature is a property of the input that the network represents, such that you can find it inside the activations. Examples: "there is a curve here", "this text is in French", "this token is part of a Python function name", "the Golden Gate Bridge".

Features as directions

Recall from Chapter 1 that a layer's activation is a vector: a point in a space with one axis per neuron. Where in that space is a feature?

The common answer is: a feature is a direction. Each feature i has an arrow W_i. For now, W_i is only a name for that arrow. It is not yet a weight of any network. If the feature is active with strength x_i, the network adds x_i·W_i to the activation. If several features are active, their arrows add up.

To read a feature, you take the dot product of the activation with that feature's direction. If the directions are orthogonal (at 90°), the reading is exact. If they are not, other features leak into the reading. This leak is interference, and it is the center of this chapter.

Three things that are easy to mix up

The equation above has three different objects in it. People often mix them up. Take a layer with 1,024 neurons.

The feature directions are like a fixed list of ingredients. The feature activations are the recipe for one input: how much of each ingredient to use. The activation vector is the finished dish.

So here are the answers to three common questions.

1. Is the activation vector a feature vector? No. The activation vector is a sum of feature vectors, each scaled by how active that feature is.
2. Does each of the 1,024 numbers give the amount of one feature? Only if every feature lines up with exactly one neuron. With superposition there are more features than neurons, so this cannot be true. The 1,024 numbers are neuron values. You get a feature's amount from a dot product with its direction.
3. Are the feature directions the columns of the layer's weight matrix? In the toy model later in this chapter, yes. In a real network, no. The next box explains.

Early evidence: word arithmetic

The first famous evidence for directions came from word embeddings. In 2013, Tomas Mikolov and colleagues trained word2vec. They found that vector arithmetic works on meanings:

The step from "man" to "woman" is about the same arrow as the step from "king" to "queen". So "gender" seems to be one direction. "Royalty" seems to be another. One caveat: the trick usually works only if you exclude the input words from the search for the nearest word.

The linear representation hypothesis

The linear representation hypothesis says that models store most concepts as directions, and that several concepts combine by adding. Note one point: the features can be very nonlinear functions of the input. "Is this a dog?" is a hard function of the pixels. Only the map from features to the activation vector is linear.

The idea in one example

Take four words: man, king, woman, queen. Suppose the model has one direction that means "royal" and another that means "female". The hypothesis says that the activation for each word is built like a recipe:

Three claims are packed into this recipe. Each one is something you can test.

1. Composition is addition. "Royal and female" is just the royal arrow plus the female arrow. Nothing special happens when they combine.
2. Intensity is length. "Very royal" is the same arrow, only longer. The direction says what; the length says how much.
3. Reading is one dot product. To ask "how royal is this?", project the activation onto the royal arrow. A single weighted sum does it. This is exactly what a linear probe does, and adding an arrow on purpose is what steering does (Chapter 7).

Try all three below. Then lower the angle between the two arrows. When the arrows are no longer perpendicular, the royal probe starts to "see" some femaleness, even though none was added. That leak is called interference, and it is the central problem of the rest of this chapter.

What would a non-linear representation look like? Imagine the model stored "queen" at some point that has nothing to do with king + woman, or stored "very royal" in a different direction from "royal". Then no single probe could read royalty, and adding an arrow would not steer. The hypothesis is a bet that this is rare. The "onion" counterexample at the end of this section shows it can happen.

Elhage et al. gave three reasons why networks would prefer linear representations:

• They are natural. A neuron computes a weighted sum, which is a template match. Template matches give linear outputs.
• They are easy to read. A neuron in the next layer can select a feature in one step, with one weighted sum.
• They generalize well. Directions let the network reuse the same "gender" arrow for many word pairs.

They also gave a ladder of properties. Each step is a stronger claim than the step before.

Park and colleagues found a subtle point. In a space with no special axes, the normal dot product is not the "right" way to measure angles. They proposed a causal inner product, based on the covariance of the unembedding vectors. (The unembedding is the matrix that turns the model's last vector into one score per word; see Chapter 4.) Under this inner product, concepts that vary independently (like gender and language) become orthogonal. Under the normal dot product, they do not.

In July 2024, Chris Olah (with editing by Adam Jermyn) added a clarification. "Linear" means mathematically linear, not "one-dimensional". It has two parts:

1. Composition is addition. A feature is present when its vector is added.
2. Intensity is scaling. A stronger feature gives a longer vector.

So a feature can be a circle or a curve and still be "linear", as long as it adds to the rest. We will see such features at the end of this chapter.

The hypothesis is a strong default, not a law. In 2024, Róbert Csordás and colleagues found a clear counterexample. Small recurrent networks that had to repeat a sequence stored each position at a different size (order of magnitude), not in a different direction. They called these "onion" representations. You cannot find them in separate linear subspaces. Larger versions of the same networks did use linear representations. The lesson: check linearity, do not assume it.

Neurons and the privileged basis

If features are directions, do the neuron axes matter at all? It depends on the layer. And the answer decides something practical: whether looking at one neuron at a time can tell you anything.

Why you should care

Almost everything in Chapter 2 worked by looking at single neurons. Pick a neuron, find the inputs that make it fire, and give it a name. That is how researchers found curve detectors, and it is how they found the strange neuron 4e:55. It is the cheapest method in interpretability, because a neuron is just one number that you can read off directly.

Now try the same method on a transformer. Most of its information travels in the residual stream (the main vector that flows through the model; Chapter 4). You pick coordinate 512, look at what makes it large, and you get a mess. Is the method broken? Did you just pick a bad coordinate?

Neither. In that layer, coordinate 512 is not a real object at all. It comes from how we chose to write the numbers down. The model itself has no way to notice it. The idea of a privileged basis tells you, before you spend a week on it, which layers have real neurons and which do not.

A layer where the axes are arbitrary

Start with a layer that only feeds into a weighted sum. Nothing else touches it.

The general rule: take any invertible matrix M. Multiply the layer's activations by M, and multiply the next layer's weights by M⁻¹. The two changes cancel, and the model computes exactly the same function. A layer where you can do this for free has a non-privileged basis. Its coordinates mean nothing on their own. Only directions survive the rotation.

Word embeddings are the classic case. Training word2vec only ever uses dot products between vectors. If you rotate every vector by the same rotation, every dot product stays the same, so the rotated model is exactly as good. The king − man + woman arithmetic still works, because all the arrows turned together. But "dimension 17" now holds completely different numbers. The gender direction is a fact about the model. Dimension 17 is not.

The residual stream behaves the same way, because the model only reads from it and writes to it with linear maps. This is why later chapters stop asking about coordinates and search for directions instead, with probes (Chapter 7) and sparse autoencoders (Chapter 8).

A layer where the axes are fixed

Now put a ReLU on the layer. ReLU looks at each coordinate separately and cuts it at zero. So it does care which axes you use. Take h = (1, −1) and compare two paths.

The results differ. In the first path, ReLU deleted the second coordinate. In the second path, the same vector had no negative coordinate, so ReLU did nothing. You can no longer rotate the layer and fix it up in the next layer's weights. The axes that ReLU acts on are built into the model. We call this a privileged basis, and its coordinates are what we call neurons. Any elementwise nonlinearity does this, GELU included.

You have already seen the evidence that this matters in real networks. In Network Dissection (Chapter 2), the authors rotated the conv5 layer of AlexNet after training. Accuracy did not change, but there were about 80% fewer unique concept detectors. If the neuron axes were arbitrary, the rotated axes should have been just as readable as the original ones. They were not. Training had lined many concepts up with the exact axes that ReLU acts on.

Which layers are which

Why "with care"

A privileged basis encourages features to align with neurons. It does not guarantee it.

The encouragement comes from how ReLU works. It can switch one neuron off without touching the others. So if a feature sits on its own neuron, the layer can cut that feature at zero by itself. If the feature is spread over several neurons, each cut hits a mixture of features. Training therefore has a reason to give a feature its own neuron when it can.

But it cannot always. If the layer needs more features than it has neurons, some neurons must carry several. Neuron 4e:55 lives in a privileged basis and still responds to cat faces and car fronts.

This is also why mixed neurons count as a puzzle. In a non-privileged basis, a coordinate that mixes many features is what you expect, because the coordinate was arbitrary to begin with. In a privileged basis, the network had a reason to keep its neurons clean and did not. Something pushed against that reason. The rest of this chapter argues that the something is superposition.

Polysemantic neurons: the evidence

A neuron is monosemantic if it responds to one clear thing. It is polysemantic if it responds to several unrelated things.

In vision models

In "Zoom In" (2020), Olah and colleagues described a neuron in the vision model InceptionV1. In their words: "InceptionV1 contains one neuron that responds to cat faces, fronts of cars, and cat legs." This is neuron 4e:55. Feature visualization shows that it is not looking for something cats and cars share. It looks for the eyes and whiskers of a cat, for furry legs, and for shiny fronts of cars. These are three different things.

Elhage et al. also note that in InceptionV1, the fraction of polysemantic neurons grows with depth. Their explanation: deeper features are more abstract and rarer. Rarer features are better candidates for superposition, as we will see.

In language models

Neurons in the first MLP layer of transformer language models are often extremely polysemantic. One reason may be that early layers must separate the same token in different contexts. For example, "die" means different things in English, German, Dutch and Afrikaans. Each of these context features is very rare.

Many language-model neurons also show a pattern: their top activations look clean, but their weaker activations are a mess of unrelated things. This "primary feature plus weak secondary features" pattern appears in the toy models too.

Polysemanticity is not the same as superposition

These two words are easy to mix up.

• Polysemanticity is an observation about a neuron: it responds to many unrelated things.
• Superposition is a hypothesis about a representation: it holds more features than it has dimensions.

Superposition causes polysemanticity in a privileged basis. But polysemanticity can have other causes. In 2023, Victor Lecomte and colleagues showed incidental polysemanticity. Random initialization can put two features on the same neuron by chance. Training then makes the overlap stronger. This can happen even when there are enough neurons for every feature.

The superposition hypothesis

Here is the hypothesis in the words of Elhage et al. (2022):

And in more detail: features are stored as almost-orthogonal directions. Because they are only almost orthogonal, one active feature looks like other features slightly active. This "noise" has a cost. But if features are sparse, the cost can be smaller than the benefit of storing many more features.

A good mental picture: the real network is a small network that is "noisily simulating" a much larger, very sparse network. The larger network is the one we want to read.

Why the world makes superposition attractive

Think about all the concepts a language model could use: every person, place, programming function, legal term, chemical, and joke format. There are millions. But any one sentence uses only a few. So each concept is sparse: it is off for almost all inputs. If two features are almost never on together, they can share space with little harm.

Two ingredients are both necessary

Superposition means accepting interference: when one feature is on, the others read small wrong values. Two things make that acceptable. A nonlinearity lets the network throw small wrong values away. Sparsity makes sure the wrong values usually are small, because features rarely fire together. Below, both are shown on the same tiny model: five features squeezed into a two-dimensional space, arranged as a pentagon. This is exactly the shape the toy model learns later in this chapter.

Ingredient 1: a nonlinearity throws interference away

Turn on feature 1 alone. The hidden vector h is just the feature 1 arrow. To read feature j, the model takes the dot product of h with arrow j. Because the arrows are not perpendicular, the four off features do not read 0. Their neighbours read +0.31 (cos 72°) and the far ones read −0.81 (cos 144°).

• In Linear mode, those wrong values go straight to the output. A bias cannot help: shifting every output by the same amount fixes nothing. This is why the best purely linear model never uses superposition. It keeps only as many features as it has dimensions (that is PCA) and drops the rest.
• In ReLU mode, every negative reading becomes 0. That removes the −0.81s for free.
• Now lower the bias to about −0.35. The small positive readings (+0.31) fall below zero too, and the ReLU removes them as well. The price: the true feature reads a bit low (0.65 instead of 1). A weak wrong signal is gone; a strong right signal survives.

Now turn on two neighbouring features, f₁ and f₂. Each one adds +0.31 of false signal to the other's neighbours, and those false signals stack up. Some now pass the bias, and the outputs go wrong. The ReLU can only clean up interference that is small. Interference stays small only if few features are on at once. That is the second ingredient.

Ingredient 2: sparsity keeps collisions rare

Let each feature be on with probability p = 1 − S, where S is the sparsity. The left panel draws 40 random inputs. Each column is one input; a dot means that feature is on. Columns where two or more features are on at once are collisions, marked in red. In a collision, the pentagon model makes errors like the ones you just saw.

The right panel compares two strategies, each with its best bias, over thousands of random inputs:

• Keep 2, drop 3 (what a linear model, i.e. PCA, does): two features get their own perpendicular directions and are always perfect. Three features are never stored. The model can only guess their average.
• Pentagon (superposition): all five are stored, with a ReLU and bias to clean up interference.

When features are dense, most inputs are collisions, and the pentagon is swamped by interference. Giving up three features entirely is the better deal. As features get sparser, collisions vanish. Then almost every input has at most one feature on, and that is exactly the case the ReLU handles well. The pentagon wins, and the gap grows with sparsity. Notice also the best bias: it becomes more negative as features get sparser, because the ReLU threshold is doing more of the work.

So the two ingredients do different jobs. The nonlinearity filters interference; sparsity keeps interference small enough to filter. Remove either one and superposition stops paying.

The next sections look at each piece more closely: how much room high-dimensional space has, and the smallest possible case of ReLU filtering.

How much room is there in high dimensions?

In d dimensions, you can fit only d directions that are exactly orthogonal. But if you allow a small error, you can fit far more. The number of directions with pairwise |cosine| below a small ε grows exponentially with d. This follows from the Johnson–Lindenstrauss lemma (1984) and from a fact called concentration of measure.

Random unit vectors in d dimensions have cosine similarities that are roughly normal with mean 0 and spread 1/√d. So in high dimensions, random directions are almost always nearly perpendicular. Try it.

Here are numbers from a quick simulation with random vectors:

In 1,000 dimensions, 10,000 random directions are all within about 10° of perpendicular. A trained network can place directions even better than random. A real model's residual stream has thousands of dimensions. So it has room for a huge number of almost-orthogonal features.

A second fact: compressed sensing

Normally, if you project a vector into fewer dimensions, you lose information. You cannot get it back. But around 2006, Candès, Romberg, Tao and Donoho showed an exception. If the vector is sparse (only a few non-zero entries), you can often recover it exactly from far fewer measurements. This field is called compressed sensing. Superposition is a cousin of it. In the toy model, the ReLU plus bias acts as a cheap, one-step decoder.

How a ReLU filters interference

Let us take the smallest possible case: two features in one dimension. The network gives feature 1 the direction +1 and feature 2 the direction −1. They point in opposite directions. This is called an antipodal pair.

To decode, the network reads h with each feature's direction, adds a bias, and applies ReLU:

Suppose only feature 1 is on, with value 0.8. Then h = 0.8. The reading for feature 2 is −0.8. This is negative interference. A ReLU turns it into 0. So the network gets the right answer for both features. The ReLU makes negative interference free. Use the demo to check this, and to see when it fails.

Three lessons come from this demo:

1. With ReLU and only one feature on, recovery is perfect. Negative interference is clipped.
2. Without ReLU (linear mode), the off feature reads as a negative number. The error is large.
3. When both features are on, they cancel inside h, and the model fails. This is the price of superposition. It is cheap only if both features are rarely on at the same time. That is why sparsity matters.

Toy Models of Superposition (2022)

Real models are too big to study superposition directly. So the authors built the smallest model where it could appear.

The setup

Imagine a large ideal model whose neurons are the features. We have n features. We ask a small model to squeeze them into m < n dimensions and then recover them.

• Data. Each input x has n features. Each feature is 0 with probability S (the sparsity). Otherwise it is a random number between 0 and 1. All features are independent.
• Importance. Each feature has an importance I_i. The loss cares more about important features. A common choice is I_i = 0.7ⁱ, so feature 1 matters most.
• Model. A matrix W of size m × n compresses: h = Wx. Then the same matrix (transposed) decompresses, a bias b is added, and a ReLU is applied.
• Loss. Importance-weighted squared error: L = Σ_i I_i (x_i − x′_i)².

Two design choices matter:

• Tied weights. Using the same W in both steps means that column W_i is "the direction of feature i". There is no confusion between a read direction and a write direction.
• The bias b. It lets the model output an average value for dropped features. More important, a negative bias removes small positive interference.

The authors compared this ReLU output model with a linear model that is the same but has no ReLU. Their main sizes were n = 20, m = 5 with I_i = 0.7ⁱ, and n = 5, m = 2 for pictures in 2D.

The basic result

• The linear model always learns the top m most important features in orthogonal directions. It ignores the rest. This is the same as PCA. Sparsity does not change this.
• The ReLU model acts like the linear model when features are dense. As sparsity rises, superposition appears. "It starts with less important features, and gradually affects the most important ones." It first forms antipodal pairs, then other shapes.

Train a toy model yourself

This widget trains the toy model in your browser with real gradient descent (Adam). There are 5 features and 2 hidden dimensions. Each arrow is one column W_i of the weight matrix: the direction where feature i lives. Each feature has its own color. Feature 1 is the most important; importance falls as I_i = rⁱ. The dashed circle marks length 1.

What you should see:

We checked these results by training the same code offline with several seeds. The paper reports the same picture. One caution from the authors: with only 2 hidden dimensions, training can get stuck in bad solutions. They trained each model several times and kept the best. If a run looks odd, press Reset and train again.

How the paper shows superposition

The authors used two pictures, which this widget also shows.

• The matrix WᵀW. Entry (i, j) is the dot product W_i·W_j. The diagonal shows how strongly each feature is stored. Off-diagonal entries are interference. A dense model has a clean diagonal block. A superposed model has many non-zero off-diagonal entries. Large negative entries mark features that point in opposite directions (for example, antipodal pairs).
• Bars per feature. Bar length is ‖W_i‖ (is the feature stored at all?). In the paper, bar color showed how much the feature shares its direction with others.

The math in one paragraph

Why does sparsity cause superposition? Split the expected loss by how many features are on at once. When S is close to 1, the most common case is "exactly one feature is on". In that case, a ReLU makes negative interference free, so the model only pays for positive interference. The problem becomes: place n points on a sphere in m dimensions so they push each other apart. This is like the Thomson problem from physics and chemistry: how to place electrons on a sphere to minimize repulsion. The pentagon is the answer for 5 points on a circle.

Superposition is a phase change

The switch to superposition is not gradual. It is sudden, like water turning to ice. The authors showed this with the simplest case: 2 features in 1 dimension. Feature 1 has importance 1. Feature 2 (the "extra" feature) has importance between 0.1 and 10. The density (1 − S) goes from 1 (always on) down to 0.01 (on 1% of the time).

For each setting, three solutions compete:

• Ignore the extra feature. W = [1, 0]. Only feature 1 is stored.
• Give the dimension to the extra feature. W = [0, 1]. Feature 1 is dropped.
• Superposition. W = [1, −1]. Both are stored as an antipodal pair. The model cannot represent both at once.

The model picks whichever solution has the lowest loss. When the loss curves of two solutions cross, the best solution jumps. This is a true first-order phase change: the slope of the best loss jumps too. The paper's training results matched the theory.

The lesson has three parts:

1. Sparsity is necessary. At high density, superposition never wins.
2. Relative importance decides who gets stored. In the dense regime, the more important feature takes the dimension.
3. The transition is sharp. A small change in sparsity can flip the whole solution.

The authors also drew a phase diagram for 3 features in 2 dimensions. There, the extra feature can become antipodal to one of the others, or two other features can pair up while the extra one gets its own dimension. Each option is a separate phase.

The geometry of superposition

The most surprising result of the paper is that superposition forms regular shapes.

Counting features

The squared size of the whole matrix, ‖W‖²_F, is a good count of stored features. Each stored feature has ‖W_i‖² ≈ 1 and each dropped feature has ≈ 0. So the average dimensions per feature is:

With all features equally important (the paper used n = 400, m = 30), the authors plotted D* against 1/(1 − S). The curve is not smooth. It is "sticky": it stays flat at D* = 1 and at D* = 1/2 for wide ranges of sparsity. The 1/2 plateau means antipodal pairs: two features per dimension. The authors note a loose resemblance to the fractional quantum Hall effect in physics.

Feature dimensionality

The authors then defined a per-feature number. It says what fraction of a dimension feature i gets:

Here Ŵ_i is the unit vector in direction W_i. The top is "how much feature i is stored". The bottom is "how many features share its space". Some values:

• A feature with its own dimension: D = 1.
• An antipodal pair: D = 1/(1 + 1) = 1/2.
• A dropped feature: D = 0.
• When features are packed well, the D_i add up to about m.

When the authors plotted every feature's D_i, the points clustered at a few sticky fractions. Each fraction is a shape. The rule: k features spread evenly in a d-dimensional subspace each get D = d/k.

These shapes are solutions to the Thomson problem that are uniform polytopes: every corner looks the same, so every feature gets the same D. Less symmetric solutions show up split into parts. For example, instead of a 5-point shape with D = 3/5, you see a triangle (2/3) plus an antipodal pair (1/2) in perpendicular subspaces. Such a combination of shapes in orthogonal subspaces is called a tegum product. Features in different parts do not interfere at all.

The authors wrote that "superposition isn't just one thing". Like ice, which has many crystal phases, superposition has many phases with different geometries.

Non-uniform features

• One feature denser than the others. Start from a pentagon (n = 5, m = 2). Make one feature denser: it pushes the others away and gets more room. Make it sparser: it gets squeezed. Past a critical point, the pentagon snaps into two antipodal pairs and the sparse feature is dropped. This is another first-order phase change.
• Correlated features (features that tend to be on together) prefer to be orthogonal to each other. If there is no room, they sit side by side. If there is still no room, they collapse into their average direction. In the dense limit, this becomes PCA. So PCA and superposition are two ends of one range.
• Anti-correlated features (only one is on at a time) prefer to share space, ideally as antipodal pairs.

Learning dynamics: energy levels

During training, single features jump between sticky fractions. Each jump matches a sudden drop in the loss. The authors suspect that the smooth loss curves of large models may be made of many such small jumps. They also saw training pass through clear geometric stages. For example, one model (6 features in 3 dimensions, in two correlated groups) first learned a PCA-like solution and then moved, in steps, to an octahedron.

Neurons, computation, and adversarial examples

Superposition in real neurons

The ReLU output model has no privileged hidden basis: you can rotate h freely. To study neurons, the authors added a ReLU on the hidden layer too: h = ReLU(Wx). Now each hidden coordinate is a neuron. With n = 10 features, m = 5 neurons and I_i = 0.75ⁱ, they saw a clear shift. When features are dense, neurons are monosemantic. As sparsity grows, neurons become polysemantic. Both kinds can exist in one model at the same time.

Computation in superposition

Storing features is one thing. Can the network also compute with them while they overlap? The authors tested a simple function: absolute value, |x| = ReLU(x) + ReLU(−x). With n = 100 sparse inputs and only m = 40 neurons (I_i = 0.8ⁱ), the model computed |x| for more features than it had neurons, once the inputs were sparse enough.

• Neurons for the most important features stayed monosemantic.
• Many neurons had one big "primary" weight plus small "secondary" weights. This matches language-model neurons whose top activations are clean but whose weaker activations are mixed.
• The authors found a motif they called asymmetric superposition. One neuron stores two features with unequal weights, so A interferes with B but not the reverse. A second neuron then inhibits the bad case.

The idea of U-AND is simple. Wire each neuron to a random subset of features. Set its threshold so it fires only when two of its inputs are on. Every pair of features shares some neurons, so every AND can be read out. The result suggests that networks can be "much bigger on the inside" than their width. It is a theoretical existence proof. It does not show that trained networks use this exact trick.

A link to adversarial examples

Without superposition, the model's output for feature 0 depends only on input feature 0. With superposition, it also depends a little on many other features: the weights look like (1, ε, −ε, ε, …). Each small ε is a lever for an attacker. Many small pushes add up.

The authors measured this. They used optimal attacks with a size limit of 0.1 × the average input norm. Vulnerability "sharply increases as superposition forms (increasing by >3×)". It tracks the number of features per dimension. Adversarial training did reduce superposition, but only with very large attacks (80% of the input norm). The authors do not claim that this is the cause of adversarial examples in real models. It is one plausible cause.

Follow-up work on superposition

The setup used 10,000 very sparse features (S = 0.999). With few training examples, the hidden vectors of the examples formed clean polygons, while the feature directions looked like a mess. With many examples, the reverse happened. Test loss went down, up, and down again as data grew: double descent. The peak sat at the crossover between the two regimes. This gives a mechanistic story for double descent. It also warns us: an overfit model may contain "features" that are really memorized data points. The authors called this work preliminary.

Gurnee and colleagues gave real evidence from language models.

• Early layers use sparse combinations of neurons. Example: the compound word "social security". No single neuron detects it, but the sum of three neurons separates it almost perfectly. Early-layer neurons also had large weights and large negative biases, as the toy model predicts.
• Middle layers have more dedicated "context neurons", for example a French-text neuron and a Go-code neuron. One neuron in Pythia-1B fires on the word "return" only in Go code.
• Scale. On average, larger models use sparser representations. But features differ. Some features get a dedicated neuron only at larger scale. Others split into several more specific neurons. Many do not change.

One limit: probing shows that information is present, not that the model uses it (see Chapter 7).

This paper links superposition to scaling laws, the smooth curves that show loss falling as models grow. In weak superposition, the loss follows a power law only if feature frequencies do. In strong superposition, loss ∝ 1/m (inverse width) for many distributions, because interference between overlapping vectors scales like 1/m. Analysis of several families of open LLMs suggests that real LLMs are in the strong regime. The authors also report that the Chinchilla scaling laws are consistent with this picture.

Two 2026 papers extend the theory. Prieto and colleagues show that when features are correlated, interference can be helpful: features that fire together can be placed so they support each other. This explains semantic clusters seen in real LLMs, which the idealized toy model cannot. Ivanov and colleagues study the eigenvalues of WWᵀ and recover the toy-model shapes (simplices, polygons, antiprisms) as special cases, with a method that applies to real weight matrices.

Beyond straight lines: circles and manifolds

The toy model assumes that each feature is one direction. Some real features are not.

Engels and colleagues used a sparse autoencoder (SAE), a tool that splits activations into many readable feature directions. Chapter 8 explains it. They grouped SAE features whose directions were similar. Some groups formed circles. The 7 days of the week sat around a circle in order. So did the 12 months. Then they tested tasks like "Two days from Monday is" (49 prompts) and "Four months from January is" (144 prompts). Mistral 7B got 31/49 and 125/144 right. Llama 3 8B got 29/49 and 143/144.

The key test was causal. They found the circle in a 5-dimensional subspace. Then they rotated the "day" around the circle by patching activations. The model's answer shifted to match. So the model does not just store a circle; it uses it to compute.

This 2025 study is a beautiful example of geometry in a real model. To predict a line break, the model must count characters on the current line, know the line width, and compare the remainder with the length of the next word.

• The character count is a curved line (a 1-D manifold). About 10 features tile it, like place cells in a mouse brain. Usually two neighbouring features are active at once.
• The top 6 directions of this curve explain 95% of its variance. In that subspace, the curve twists like a helix.
• The curve "ripples". The authors explain this as a superposition effect inside the manifold: the best low-dimensional way to make each count similar to its neighbours and different from far counts.
• Special "boundary" attention heads (the parts of a transformer that move information between tokens; Chapter 4) rotate the count manifold so that count i lines up with line width k ≈ i. The head then attends to the last newline when the line is nearly full.
• At the end, "characters remaining" and "next word length" sit in near-orthogonal subspaces. This makes the "break or not" decision a simple linear cut.

The authors stress a duality. You can describe the same computation as discrete features and circuits, or as continuous curves and rotations. Both views are correct. Some inserted character sequences even caused "visual illusions" that distorted the model's sense of position.

Why this matters: three ways out

Superposition is the main obstacle to reading networks. Elhage et al. explained why:

• We would like to list all features. Then we could make strong claims like "no deception feature takes part in this circuit". Without superposition, features would be neurons, and listing neurons is easy. Superposition breaks this.
• Simple tools become unreliable. Cosine similarity can mislead, because unrelated features get small positive dot products.
• You cannot find one direction, remove it, and repeat, because the directions are not orthogonal.

The paper proposed three ways out.

1. Build models without superposition. For example, add a penalty that makes activations sparse, or use mixture-of-experts layers so that each neuron can be rare without cost.
2. Find an overcomplete basis after training. Collect many activations H. Find a sparse matrix A and a dictionary B with H ≈ A·B, where B has many more directions than the layer has dimensions. The authors called this "a relatively standard sparse coding (or dictionary learning) problem". They expected hard problems: the number of features is unknown, and the dictionary might need to be "a thousand or more times larger" than the layer.
3. Hybrid approaches. Reduce superposition during training, then decode what remains.

Path 2 became the main road. In 2023, Anthropic's "Towards Monosemanticity" trained sparse autoencoders on a small one-layer transformer and found thousands of mostly interpretable features. Chapter 8 tells that story.

References

1. Elhage, N., Hume, T., Olsson, C., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. arXiv:2209.10652.
2. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., Carter, S. (2020). Zoom In: An Introduction to Circuits. Distill.
3. Mikolov, T., et al. (2013). Distributed representations of words and phrases and their compositionality (word2vec). NeurIPS.
4. Park, K., Choe, Y. J., Veitch, V. (2023). The Linear Representation Hypothesis and the Geometry of Large Language Models. ICML 2024.
5. Olah, C. (edited by A. Jermyn) (2024). What is a Linear Representation? What is a Multidimensional Feature? Transformer Circuits, July 2024 update.
6. Csordás, R., Potts, C., Manning, C. D., Geiger, A. (2024). Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations. BlackboxNLP 2024.
7. Lecomte, V., Thaman, K., Schaeffer, R., Bashkansky, N., Chow, T., Koyejo, S. (2023). What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes.
8. Henighan, T., Carter, S., Hume, T., et al. (2023). Superposition, Memorization, and Double Descent. Transformer Circuits Thread.
9. Scherlis, A., Sachan, K., Jermyn, A., Benton, J., Shlegeris, B. (2022). Polysemanticity and Capacity in Neural Networks.
10. Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., Bertsimas, D. (2023). Finding Neurons in a Haystack: Case Studies with Sparse Probing.
11. Hänni, K., Mendel, J., Vaintrob, D., Chan, L. (2024). Mathematical Models of Computation in Superposition.
12. Braun, D., Bushnaq, L., Heimersheim, S., Mendel, J., Sharkey, L. (2025). Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition. (Source of the "compressed computation" toy model.)
13. Bhagat, J., Molas-Medina, S., Giglemiani, G., Heimersheim, S. (2025; arXiv 2026). Compressed Computation is (probably) not Computation in Superposition. LessWrong; arXiv:2606.14673.
14. Ferreira da Silva, F., Heimersheim, S. (2026). Compressed Computation under L⁴ Loss is likely Computation in Superposition.
15. Engels, J., Michaud, E. J., Liao, I., Gurnee, W., Tegmark, M. (2024). Not All Language Model Features Are One-Dimensionally Linear. ICLR 2025.
16. Li, Y., Michaud, E. J., Baek, D. D., Engels, J., Sun, X., Tegmark, M. (2024). The Geometry of Concepts: Sparse Autoencoder Feature Structure.
17. Gurnee, W., Ameisen, E., Kauvar, I., Tarng, J., Pearce, A., Olah, C., Batson, J. (2025). When Models Manipulate Manifolds: The Geometry of a Counting Task. Transformer Circuits Thread.
18. Modell, A., Rubin-Delanchy, P., Whiteley, N. (2025). The Origins of Representation Manifolds in Large Language Models.
19. Liu, Y., Liu, Z., Gore, J. (2025). Superposition Yields Robust Neural Scaling. NeurIPS 2025.
20. Prieto, L., Stevinson, E., Barsbey, M., Birdal, T., Mediano, P. A. M. (2026). From Data Statistics to Feature Geometry: How Correlations Shape Superposition.
21. Ivanov, G., Oozeer, N., Raval, S., Pejovic, T., Upadhyay, S., Abdullah, A. (2026). Spectral Superposition: A Theory of Feature Geometry.
22. Johnson, W. B., Lindenstrauss, J. (1984). Extensions of Lipschitz mappings into a Hilbert space. (The Johnson–Lindenstrauss lemma.)
23. Candès, E., Romberg, J., Tao, T. (2006); Donoho, D. (2006). Foundational papers on compressed sensing.