The birth: looking inside vision models

Interpretability grew up on image models. First, people asked which pixels mattered. Then they asked what each neuron wants to see. Finally, they read whole algorithms from the weights. This chapter follows that path from 2009 to 2022.

The big picture: from "where" to "how"

Deep learning took off in 2012, when AlexNet won the ImageNet image competition. Soon, researchers wanted to know what these networks had learned. Images are a good place to start. You can look at an image and see what it shows. You can draw a picture of what a neuron responds to. So the first interpretability tools were built for vision models.

The work in this era asked three questions, one after the other:

1. Which input pixels mattered for this output? This is attribution. It makes a heatmap over the input.
2. What does this neuron detect? This is feature visualization and unit labeling.
3. How does the network compute its features from earlier features? This is the circuits approach. It is the start of mechanistic interpretability.

Each step goes deeper. Attribution looks only at the input and the output. Feature visualization looks at single parts inside. Circuits look at the parts and the weights that connect them. Only the last step gives an algorithm.

Which pixels mattered? Saliency and attribution

The first popular tools were saliency maps. A saliency map is a heatmap over the input image. Bright pixels are pixels that "mattered" for the output.

Gradient saliency (2013)

Karen Simonyan, Andrea Vedaldi and Andrew Zisserman at Oxford had a simple idea. Take the score for one class, for example "cat". Compute the gradient of that score with respect to each input pixel. The gradient tells you how much the score changes if you change that pixel a little. Large gradient means a sensitive pixel. This needs only one backward pass through the network.

Plain gradients look noisy and speckled. They also saturate: a pixel can be very important but have a near-zero gradient, because the network is already "sure". So many variants followed.

Why a heatmap is not a mechanism

Attribution is useful. But it answers a narrow question. It tells you which inputs moved the output for one example. It does not tell you what the network computed in between. It does not name any internal variable. It does not give an algorithm.

The sanity check that shook attribution

Saliency maps looked convincing. But were they really about the model? In 2018, Julius Adebayo, Been Kim and colleagues tested this with a simple idea. If a heatmap explains a model, it must change when the model changes.

They ran two tests:

1. Model randomization. Replace the trained weights with random weights, layer by layer from the top down. A random model has learned nothing. So a faithful explanation should become different, or become noise.
2. Data randomization. Retrain the model on the same images, but with shuffled labels. The model can only memorize. Again, a faithful explanation should change.

The result was a shock. Some methods, especially guided backpropagation and guided Grad-CAM, barely changed. Their maps looked like the output of a plain edge detector run on the image. In the authors' words, some saliency methods are "independent both of the model and of the data generating process". Plain gradients and Grad-CAM passed the test better.

A related paper by Kindermans and colleagues (2017) found another problem. Add a constant shift to every input pixel. Then change the first layer's bias to cancel the shift. The new model gives exactly the same outputs as the old one. But several attribution maps did change. So these maps can change when the model's behaviour does not.

What does this neuron want to see? Feature visualization

Attribution starts from one input. Feature visualization starts from one neuron. It asks: what input makes this neuron fire the most?

The core trick: optimize the input

In 2009, Dumitru Erhan, Yoshua Bengio and colleagues at Montreal had the core idea. Normally, training changes the weights to reduce an error. Here, you freeze the weights and change the input image instead. You do gradient ascent on the pixels to make one neuron's activation as large as possible. They called it activation maximization. They used it on small deep networks trained on handwritten digits and on patches of natural images. Before this, people could only picture first-layer units, by drawing their weights. This method let them see what units in higher layers respond to.

On large ImageNet networks, the naive method fails. The result is a mess of high-frequency noise. The network responds strongly to strange pixel patterns that no real photo contains. These patterns are close cousins of adversarial examples: inputs with tiny, invisible changes that make a model give a wrong answer.

DeepDream (2015)

In June 2015, Alexander Mordvintsev, Chris Olah and Mike Tyka at Google posted "Inceptionism". They started from a real photo. Then they amplified whatever a layer already detected: "whatever you see there, I want more of it." Clouds turned into dog faces. Leaves grew eyes and pagodas. The images went viral. For most people, this was the first look inside a neural network. It was art more than science, but it started the collaboration that led to the next papers.

Feature Visualization on Distill (2017)

In November 2017, Chris Olah, Alexander Mordvintsev and Ludwig Schubert published "Feature Visualization" in the online journal Distill. They studied InceptionV1 (also called GoogLeNet), an ImageNet model. The article became a practical guide. Its main lessons:

• Regularize. Without help, optimization finds noise. With transformation robustness (randomly jitter, rotate and scale the image at each step), the image must work under small changes. This removes most noise.
• Precondition. Optimize in a decorrelated, Fourier-transformed color space (a way to write the image as a sum of smooth waves). This removes high-frequency artifacts. Their recipe: 2560 steps of Adam at learning rate 0.05, with jitter up to 16 pixels.
• Optimize logits, not probabilities. Logits are the raw class scores; the softmax function turns them into probabilities. The easiest way to raise a softmax probability is to suppress other classes. That gives worse images.
• Show diversity. Optimize several images at once with a penalty for being similar. This shows the different "facets" of a neuron.
• Use dataset examples too. Real images that strongly activate the neuron show what fires it in practice. Optimized images isolate the cause. You need both.

Our toy neuron is much simpler than a real one. But it shows the main effect. Without regularization, the neuron only changes the pixels it looks at. The rest stays as noise, and the picture is hard to read. With regularization, the noise goes away and a clean curve appears. In real networks the effect is stronger, because real neurons also respond to strange high-frequency patterns.

Look also at the activation number. Without regularization, it climbs much higher. The unregularized image is "better" for the neuron, but worse for a human reader. Regularization trades some raw activation for an image that we can understand.

An early polysemantic neuron

The 2017 article also showed a puzzle. Neuron 55 in layer mixed4e (written 4e:55) responds to cat faces, fox faces, and the fronts of cars. The authors wrote that such examples suggest "neurons are not necessarily the right semantic units for understanding neural nets." This is polysemanticity. It will become one of the central problems of the field.

Building Blocks (2018) and Activation Atlas (2019)

"The Building Blocks of Interpretability" (Olah and colleagues, March 2018) combined feature visualization, attribution and dimensionality reduction into interfaces. The key object is the activation cube: at each layer, activations have an x position, a y position and a channel. You can slice the cube by neuron, by spatial position, or by channel. At each image position, they showed the activation vector as small feature-visualization icons. They called this a semantic dictionary. They also measured attribution between layers, not only to pixels. For example, a "floppy ear" feature helped a "Labrador retriever" decision.

"Activation Atlas" (Shan Carter and colleagues, March 2019) asked a bigger question. Single neurons show only a few directions. What does the whole space of a layer look like? They collected activations from one million ImageNet images. They laid these vectors out in 2D with UMAP, averaged the vectors in each grid cell, and drew each average. The result is a map of concepts: buildings, food, animal ears, water.

The atlas also exposed weak spots. In the "grey whale" vs "great white shark" region, they saw a baseball-like feature. They pasted a small baseball onto a whale photo. The model changed its answer to "great white shark". The patch was found by reading the model, not by blind search. This was a small early sign that understanding can predict behaviour.

Counting interpretable neurons: Network Dissection

Feature visualization gives pictures. But a person must look at each picture and guess a label. That does not scale, and it is not objective. In 2017, David Bau, Bolei Zhou and colleagues at MIT asked: can we label every neuron automatically, and count how many are interpretable?

How it works

They built a dataset called Broden ("Broadly and Densely Labeled"). In Broden, every pixel has labels from six categories: scene (468 classes), object (584), part (234), material (32), texture (47) and color (11). For each convolution unit, they did this:

1. Run the unit over all images. Find a threshold so that only the top 0.5% of its activations are above it.
2. For each image, upsample the unit's activation map to image size. Mark the pixels above the threshold. This gives a binary mask.
3. Compare the unit's mask with the mask of each concept, for example "lamp". Compute the IoU (intersection over union): the overlap divided by the total area covered by either mask.
4. If IoU is larger than 0.04 over the dataset, the unit is a detector for that concept. The best concept is its label.

What they found

• They tested AlexNet, GoogLeNet, VGG-16 and ResNet-152. Many units matched concepts. Deeper layers had more object and part detectors.
• Humans agreed with many labels. For AlexNet trained on Places205, the fraction of units that people rated as interpretable rose from 57 of 96 in conv1 to 194 of 256 in conv5.
• Self-supervised models had fewer object detectors than supervised models.

Net2Vec, GAN Dissection and the role of units

Net2Vec (Ruth Fong and Andrea Vedaldi, 2018) asked: what if a concept lives in several units together? They learned a weight vector over all filters for each concept. They found that "multiple filters are required to code for a concept", and that filters "are not concept specific". There are more concepts than channels, so the network must use combinations. This points directly at superposition.

GAN Dissection (Bau and colleagues, 2019) turned dissection into an experiment. They studied a Progressive GAN that generates pictures of churches. Some units in layer 4 matched "tree". When they switched off 5, then 10, then 20 tree units, trees vanished from the pictures while buildings stayed. When they switched on door units, doors appeared, but only where a door makes sense: on a wall, not in the sky. This was ablation and insertion used as causal tests. These are the same kinds of tests that Chapter 5 covers for language models.

In 2020, Bau and colleagues summarized this line of work in PNAS. In a VGG-16 trained to classify scenes, a few units carried a whole class. For the class "ski resort", four units mattered most. They detected snow, mountains, trees and houses. Removing those four units together dropped the accuracy for that class from 81% to 64%. Removing the 20 most important units dropped it to 54%, near chance (50%). But removing the 492 least important units, and keeping only those 20, left it at 78%.

Concepts as directions: TCAV

Network Dissection starts from units. Been Kim and colleagues at Google started from the other side: a human concept. Their method, TCAV (2018), asks: does the concept "striped" matter to the model's "zebra" output?

1. Collect images of the concept (striped things) and some random images.
2. Run both sets through the model. Record the activations at one layer.
3. Train a linear classifier to separate "striped" from "random" activations. The vector perpendicular to its decision line is the concept activation vector (CAV). It is a direction in activation space.
4. For each zebra image, ask: if we move the activation a little along the CAV, does the zebra score go up? This is a directional derivative.
5. The TCAV score is the fraction of zebra images where the answer is yes.

A random image set always gives some direction. So the authors repeated the test against many random sets (typically 500) and used a statistical test. A concept only counts if its score is reliably different from chance. The results matched intuition: "red" mattered for "fire engine", "striped" for "zebra", and "Siberian husky" for "dogsled". They also found biased links between gender or race concepts and some classes. And they applied it to a medical model that grades diabetic eye disease, checking concepts with a doctor.

Early neurons in text models

Vision was not the only place where single units looked meaningful. Two results on text models were famous.

Karpathy's readable LSTM cells (2015)

Andrej Karpathy, Justin Johnson and Fei-Fei Li trained character-level recurrent networks (LSTMs) on Tolstoy's War and Peace and on the Linux kernel source code. Some single cells did clear jobs:

• One cell counted line length. It started high and slowly decayed with each character until the next newline.
• Some cells turned on inside quotes, inside comments, or inside the condition of an if statement.
• One cell tracked the depth of indentation in code.

Karpathy's blog estimated that only about 5% of cells were this clear. Most cells were hard to read.

The sentiment neuron (2017)

Alec Radford, Rafal Jozefowicz and Ilya Sutskever at OpenAI trained a single-layer LSTM (a "multiplicative" LSTM) with 4,096 units. Its only task was to predict the next byte of Amazon product reviews (82 million reviews). It never saw a sentiment label. Yet one single unit tracked sentiment. A threshold on that one unit classified IMDB movie reviews with 92.30% accuracy. Fixing the unit to a high or low value made the model write positive or negative reviews. This is an early example of reading and steering a model through one internal variable.

These results gave hope that neurons might often be clean, human-readable variables. As we will see, the truth is more mixed. Clean single neurons exist, but many neurons are polysemantic.

Zoom In: the Circuits program begins

By 2020, Chris Olah's team had moved to OpenAI. The team (Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter, Chelsea Voss and others) started a new kind of project on Distill: the Circuits thread. It was a series of short articles. The work was discussed in the open, in a #circuits channel on the Distill Slack that anyone could join.

The first article, "Zoom In: An Introduction to Circuits" (March 2020), made a bold proposal. Most interpretability work tries to explain a whole network at once, with a simple summary. What if we did the opposite? What if we "zoomed in" and spent thousands of hours tracing single neurons and their connections, like a biologist with a microscope?

The three claims

The article modeled its claims on Theodor Schwann's 1839 claims about cells. The authors stated them as "speculative" and open to being wrong:

1. Features: "Features are the fundamental unit of neural networks. They correspond to directions. These features can be rigorously studied and understood."
2. Circuits: "Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood."
3. Universality: "Analogous features and circuits form across models and tasks."

The model under the microscope was InceptionV1. Its layers have names: three plain convolution layers (conv2d0, conv2d1, conv2d2), then "mixed" Inception blocks (mixed3a, mixed3b, mixed4a to mixed4e, mixed5a, mixed5b). A neuron is written layer:index. For example, 3b:379 is neuron 379 in mixed3b.

How do you prove a neuron is a curve detector?

"Zoom In" gave seven separate lines of evidence for the curve detectors in mixed3b. This list became a model for how to make a careful claim about a feature:

1. Feature visualization: optimized images show curves.
2. Dataset examples: the real images that fire the neuron most contain curves.
3. Synthetic examples: drawn curves of the right orientation make it fire.
4. Joint tuning: as you rotate a curve, firing hands off smoothly from one curve detector to the next.
5. Feature implementation: you can read from the weights how the neuron builds curves from earlier lines and curves.
6. Feature use: later neurons (circles, spirals) use the curve detectors as inputs in sensible ways.
7. Handwritten circuits: you can write weights by hand that do the same job.

Curve detectors and curve circuits

The Circuits team spent most effort on one family of features: curve detectors. A dedicated article, "Curve Detectors" (Nick Cammarata and colleagues, June 2020), studied the 10 curve neurons in mixed3b. There are four curve families in InceptionV1: in conv2d2 (11 neurons, "tiny curves"), mixed3a (11), mixed3b (10) and mixed4a (12).

Each curve detector prefers one orientation. Together, the family covers all orientations around the circle. They respond to curves over a wide range of sizes (radii). They ignore "cosmetic" details like brightness, texture and color. And they are sparse: they fire at only about 10% of image positions, mostly weakly. A strong response means a real curve of the right shape.

Scientists who study the brain describe a neuron by its tuning curve: its response as you change one property of the stimulus. The Circuits team borrowed this tool. Try it below.

How the network builds a curve detector

"Curve Circuits" (January 2021) went one step further. It asked: how are curve detectors computed from the layer below? The team read the weights.

The curve detectors in mixed3b get inputs from early curve detectors and line detectors in mixed3a, through a 5×5 convolution. When you plot those weights as a 5×5 grid, you see a picture:

• Early curves of the same orientation have a band of positive weights shaped like a curve. The detector wants a small curve at each point along a bigger curve.
• Early curves of the opposite orientation have negative weights. The detector is inhibited by curves bent the wrong way.
• When the detector's orientation rotates, the weight pattern rotates with it. The team called this an equivariant circuit.

The article summed up the result in one sentence: "although curve detection involves more than 50,000 parameters, those parameters actually implement a simple algorithm that can be read off the weights and described in just a few English sentences."

Thinking in families made this possible. It is much easier to think about "the line family in conv2d2" than about 33 separate line neurons.

The artificial artificial neural network

Then the team did something new. They wrote the weights of a curve detector by hand, in Python, from their understanding. They called it an "artificial artificial neural network". Their handmade detectors responded to curves much like the real ones, over a range of radii and orientations. The hand-set weights looked like a cleaner version of the learned pattern.

Dog heads, cars, and the first sign of superposition

"Zoom In" also showed circuits for more complex features. Two of them taught lessons that last to this day.

Dog heads: "unioning over cases"

A dog head can face left or right. Over about four layers, InceptionV1 builds two separate paths. One path detects left-facing dog heads. The other detects right-facing dog heads. The two paths inhibit each other. Then both feed with positive weights into units that detect a dog head in any direction. The team called this motif unioning over cases. They wrote: "We're somewhat surprised that gradient descent could learn to do this!"

Cars in superposition

In mixed4c, there is a neuron that detects cars. Its weights look for windows at the top of its 5×5 window and wheels at the bottom. This is an easy-to-read circuit: a car is windows above wheels.

But then something odd happens. In the next layer, the model does not build a pure car detector. Instead, it "spreads its car feature over a number of neurons that seem to primarily be doing something else — in particular, dog detectors."

Why would a network do this? The authors gave an answer and a name:

Read this carefully, because it contains the whole theory of Chapter 3 in a few lines. A layer with n neurons can only hold n fully independent directions. But it can hold many more directions that are almost independent. If two features rarely appear together (cars and dogs in the same spot), the network can share neurons between them with little harm.

Polysemantic neurons

Superposition explains the puzzle of neuron 4e:55. "Zoom In" described it as "a polysemantic neuron which responds to cat faces, fronts of cars, and cat legs." It is not detecting one shared visual pattern. It looks for cat eyes and whiskers, for furry legs, and for shiny car fronts.

Why is this a problem? "Zoom In" put it simply: "If one neuron with five different meanings connects to another neuron with five different meanings, that's effectively 25 connections that can't be considered individually." Circuit analysis depends on clean features. Polysemantic neurons break it. The authors hoped one day to "unfold" a network into pure features. That hope led, years later, to the sparse autoencoders of Chapter 8.

A catalog of early vision: families, symmetry, and surprises

Every neuron in the first five layers

"An Overview of Early Vision in InceptionV1" (April 2020) did something no one had done before. It looked at every neuron in the first five layers, about 1,056 neurons, and sorted them into families, like a periodic table.

The authors were honest about limits. The families are "ad-hoc, human defined", and their understanding of many neurons was "low-confidence". But the catalog shows a clear trend: from simple oriented edges and colors to curves, shapes and the first parts of heads.

Equivariance: the same feature, transformed

Many neurons are transformed copies of each other. "Naturally Occurring Equivariance in Neural Networks" (December 2020) cataloged the kinds:

• Rotation: curve and line detectors at many angles.
• Scale: small circle detectors early, larger ones later.
• Hue: color center-surround units for different colors.
• Reflection: left-facing and right-facing pairs, common in the middle layers.
• Others: short-snout vs long-snout dog heads, and human vs dog versions of a feature.

The symmetry also appears in the weights: when the feature rotates, its weight pattern rotates too. The authors estimated that "perhaps 80% of features in early vision" are rotationally equivariant or invariant. But early vision is only about 10% of InceptionV1, and rotational equivariance is rare in the later layers. Engineers had designed networks with built-in symmetry before (group-equivariant CNNs). Here the network learned the symmetry by itself.

High-low frequency detectors: a feature nobody expected

Fifteen neurons in mixed3a detect high spatial frequency on one side and low frequency on the other. Picture a sharp, detailed object in front of a blurry background. The edge between them has fine texture on one side and smooth blur on the other. These neurons help to find such object boundaries. They are not boundary detectors themselves, because a change in frequency can also occur without an object edge. Their main use is as inputs to boundary detectors in mixed3b.

The authors wrote that this was "not a feature that we would have expected a priori to find." The team showed it with feature visualization, dataset examples, synthetic tuning curves and the weights. This was a small but real case of learning a new idea from a neural network.

Visualizing weights

"Visualizing Weights" (February 2021) explained the practical tools behind all this. It opened with the analogy that defines the field: "The problem of understanding a neural network is a little bit like reverse engineering a large compiled binary of a computer program." Its tools include:

• Show a convolution weight as a small 2D grid of colored cells, with one color for positive weights and another for negative weights.
• Always show the input and output features next to the weights, so the numbers have meaning.
• Use expanded weights: multiply weights through intermediate layers to get the effective connection between two neurons that are not neighbors. (This ignores or simplifies the nonlinearities in between.)

Large-scale structure: branches and bands

The last articles in the thread looked for larger structure, like organs in a body.

Branch specialization (April 2021): when a layer splits into parallel branches, similar features cluster in the same branch. AlexNet (2012) was split over two GPUs for engineering reasons. One GPU learned black-and-white Gabor filters. The other learned low-frequency color detectors. In InceptionV1, all 30 curve-related features in mixed3b sit in its 5×5 branch, which holds only 96 of its 480 neurons. The chance of this happening at random is less than 1 in 10²⁰.

Weight banding (April 2021): in the last convolution layer of several ImageNet models, the weights show horizontal stripes. The stripes appear in models that use global average pooling and not in AlexNet, which does not. The team linked the bands to a way of keeping track of vertical position after pooling throws location away.

Universality: first evidence

The third claim of "Zoom In" was universality. The team found curve detectors and high-low frequency detectors not only in InceptionV1 but also in AlexNet, VGG19 and ResNetV2-50, and in models trained on a different dataset (Places365). They asked if there might be a "periodic table of low-level visual features". In their own words, the evidence was still "anecdotal", but it was encouraging.

Multimodal neurons in CLIP

In March 2021, Gabriel Goh and the team turned the microscope on CLIP. CLIP is an OpenAI model that learns from images paired with text captions. It has an image side and a text side. The team studied neurons near the end of the image side of a ResNet model called RN50-x4.

They had a famous result from brain science in mind. In 2005, Quian Quiroga and colleagues found single neurons in the human brain that respond to one person, such as Halle Berry. The neuron fires for photos of her, drawings of her, and even her written name. CLIP had neurons like that.

• The Spider-Man neuron (neuron 244) fires for photos of Spider-Man, comic drawings, the costume, and the written word "spider".
• Person neurons for well-known people. One public figure had a neuron in every CLIP model they checked.
• Region neurons, from hemispheres down to single countries and cities. About 4% of the neurons were regional.
• Emotion neurons. Complex emotions looked like sums of simpler ones. For example, "jealousy" ≈ "success" + "grumpy".

The team sampled 50 neurons from this layer. 76% were interpretable, 18% were polysemantic but with readable facets, and 6% were not readable.

Typographic attacks

Because CLIP's neurons respond to written words, text can hijack its decisions. The famous example: a Granny Smith apple with a paper label that says "iPod". CLIP classified the picture as an iPod. In an automated test, the best attacks succeeded 97% of the time while changing only about 7% of the image's pixels. The team found some attack words by reading neurons. For example, a neuron for "piggy bank" also responded to "$$$" and prices.

Mechanistic interpretability gets its name

The phrase "mechanistic interpretability" comes from Chris Olah and the Circuits group, around 2020–2021. "Zoom In" already talked about understanding "the mechanistic implementations of neurons in terms of their weights". After the vision thread ended in April 2021, much of the team moved to Anthropic. There, the work continued as the Transformer Circuits thread, starting in December 2021 (Chapter 4).

In June 2022, Olah wrote a short essay that set out the idea clearly: "Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases". It opens:

The essay makes three main points:

1. The curse of dimensionality. You cannot understand a function over a huge input space by testing inputs one by one. Programs are understandable because "the code gives a non-exponential description of the program's behavior". For networks, the weights are that short description.
2. Variables come first. The line y = x + 5 means nothing unless you know what x and y are. So the central task is to give meaning to the "memory locations" of the network. That means finding its features.
3. Interpretable bases matter. Nonlinearities like ReLU push features to line up with neurons, like a CPU that works on bytes. Polysemantic neurons break this, and make reverse engineering much harder.

Olah also asked: what is a feature? He listed candidate answers. A feature might be a property that humans can name, but that is too human-centered. It might be "the things a network would ideally dedicate a neuron to if you gave it enough neurons". This last definition links features directly to superposition. It is the definition that Chapter 3 builds on.

What critics said

This era had strong critics. Their points shaped the more careful methods that came later.

What came next

By 2021, the vision era had built the main ideas of the field:

• Features as directions, often aligned with neurons, sometimes not.
• Circuits as features connected by weights that you can read.
• Motifs that repeat: equivariance, unioning over cases, inhibition.
• Causal tests: ablation and insertion, as in GAN Dissection.
• Concept directions: TCAV and linear classifiers on activations.
• Two open problems: polysemanticity and superposition.

Two paths led forward. The first path asked: why are neurons polysemantic, and can we undo it? That path leads to Chapter 3 (superposition) and later to Chapter 8 (sparse autoencoders). The second path took the circuits approach to a new kind of model, the transformer, which was taking over AI. That path leads to Chapter 4. Language models have no pictures to visualize, so the field had to invent new tools. But the core questions stayed the same: what are the features, and how are they connected?

References

1. Erhan, Bengio, Courville, Vincent (2009). Visualizing Higher-Layer Features of a Deep Network. Univ. of Montreal Tech. Report 1341.
2. Zeiler, Fergus (2013). Visualizing and Understanding Convolutional Networks. ECCV 2014.
3. Simonyan, Vedaldi, Zisserman (2013). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.
4. Szegedy et al. (2013). Intriguing properties of neural networks.
5. Springenberg, Dosovitskiy, Brox, Riedmiller (2014). Striving for Simplicity: The All Convolutional Net.
6. Mordvintsev, Olah, Tyka (2015). Inceptionism: Going Deeper into Neural Networks. Google Research Blog.
7. Karpathy, Johnson, Fei-Fei (2015). Visualizing and Understanding Recurrent Networks.
8. Ribeiro, Singh, Guestrin (2016). "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD.
9. Selvaraju et al. (2016). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV 2017.
10. Sundararajan, Taly, Yan (2017). Axiomatic Attribution for Deep Networks. ICML.
11. Lundberg, Lee (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS.
12. Bau, Zhou, Khosla, Oliva, Torralba (2017). Network Dissection: Quantifying Interpretability of Deep Visual Representations. CVPR.
13. Radford, Jozefowicz, Sutskever (2017). Learning to Generate Reviews and Discovering Sentiment.
14. Olah, Mordvintsev, Schubert (2017). Feature Visualization. Distill.
15. Kindermans et al. (2017). The (Un)reliability of Saliency Methods.
16. Kim et al. (2018). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML.
17. Fong, Vedaldi (2018). Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks. CVPR.
18. Olah et al. (2018). The Building Blocks of Interpretability. Distill.
19. Morcos, Barrett, Rabinowitz, Botvinick (2018). On the importance of single directions for generalization. ICLR.
20. Adebayo et al. (2018). Sanity Checks for Saliency Maps. NeurIPS.
21. Bau et al. (2019). GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. ICLR.
22. Carter, Armstrong, Schubert, Johnson, Olah (2019). Activation Atlas. Distill.
23. Olah et al. (2020). Zoom In: An Introduction to Circuits. Distill.
24. Olah et al. (2020). An Overview of Early Vision in InceptionV1. Distill.
25. Cammarata et al. (2020). Curve Detectors. Distill.
26. Bau et al. (2020). Understanding the role of individual units in a deep neural network. PNAS. (arXiv version)
27. Olah et al. (2020). Naturally Occurring Equivariance in Neural Networks. Distill.
28. Schubert et al. (2021). High-Low Frequency Detectors. Distill.
29. Cammarata et al. (2021). Curve Circuits. Distill.
30. Voss et al. (2021). Visualizing Weights. Distill.
31. Goh et al. (2021). Multimodal Neurons in Artificial Neural Networks. Distill.
32. Voss et al. (2021). Branch Specialization. Distill.
33. Petrov et al. (2021). Weight Banding. Distill.
34. Olah (2022). Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases. Transformer Circuits Thread.
35. Geirhos, Zimmermann, Bilodeau, Brendel, Kim (2023). Don't trust your eyes: on the (un)reliability of feature visualizations.
36. Gorton (2024). The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision.
37. Borowski et al. (2021). Exemplary natural images explain CNN activations better than state-of-the-art feature visualization. ICLR.
38. Rudin (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence.