Sparse autoencoders for interpretability
Dictionary-learning models trained on a network's activations to split them into sparse, often human-readable features; a main tool for finding units of computation in language models, with disputed practical value.
Feature dictionaries from sparse autoencoders underlie much current interpretability work, from circuit tracing to attempts at steering and monitoring model behaviour.
A way to split a network's internal numbers into a long list of mostly-quiet, often readable features.
- 1Neurons mix many concepts
- 2Train helper to rebuild activations
- 3Force it to use few units
- 4Units become features, but unstable
The question. Inside a language model, each token is represented at each layer by a long vector of numbers, its activations. Interpretability asks what those numbers mean. The obvious idea, that each entry (each "neuron") stands for one concept, mostly fails: a single neuron often responds to several unrelated things.
Superposition. One explanation: the model stores more concepts (features) than it has dimensions by giving each a direction in the vector space rather than an axis. In 3D only three directions can be exactly perpendicular, but in a 4096-dimensional space very many directions can be almost perpendicular. If each concept is rarely present, a few active ones can share the space with little interference. The activation vector is then roughly
Go deeper
The full technical text.
Motivation: superposition
Individual neurons in a transformer often respond to several unrelated things. One hypothesis is superposition: the network represents more features than it has dimensions by assigning them nearly orthogonal directions, relying on the fact that each feature is rarely active. If so, the natural unit of analysis is a direction in activation space, not a neuron, and the directions must be found.
The method
A sparse autoencoder (SAE) is trained to reconstruct a model's activations (the residual stream, an MLP output or an attention output at a chosen layer) through a wide hidden layer whose activations are pushed to be sparse. Each hidden unit's decoder column is a candidate feature direction. Cunningham et al. (2023) and Bricken et al. (Anthropic, 2023) showed that the features of such dictionaries are more interpretable by automated and human ratings than neurons, in small models. Templeton et al. (Anthropic, 2024) scaled the approach to a production-scale model. Gao et al. (OpenAI, 2024) introduced the TopK activation, which sets sparsity directly, and studied scaling laws of SAE size and sparsity, training an SAE with 16 million latents on GPT-4 activations. DeepMind released Gemma Scope (Lieberum et al., 2024), a suite of open SAEs on Gemma 2 models.
Evaluation and doubts
SAEs are evaluated by reconstruction fidelity versus sparsity, by automated and human interpretability scores, and by downstream tasks. SAEBench (Karvonen et al., ICML 2025) packages eight metrics, covers over 200 open SAEs across eight architectures, and reports that gains on proxy metrics do not reliably translate to better practical performance. The Mechanistic Interpretability Benchmark (Mueller et al., ICML 2025) reports that SAE features were not better than neurons in its causal-variable localization track. A Google DeepMind mechanistic-interpretability team post (26 March 2025) reported that SAE probes underperformed dense linear probes at out-of-distribution harmful-intent detection, and that the team was deprioritising fundamental SAE research as a result. Paulo and Belrose (2025, preprint) reported that SAEs trained on the same data with different seeds learn partly different features (for example only about 30% shared at 131K latents on a Llama 3 8B layer), which complicates treating the dictionary as discovering 'the' features.
SAEs are therefore both a widely used tool and a contested one; the linked problems ask for measurements that would settle two of the disputes.
Words on this page (11)
- sparse autoencoder
- A model trained to rebuild a network's internal numbers using only a few active parts at a time, to find readable features.
- feature
- A direction in a network's internal numbers that seems to stand for a human-understandable concept.
- activation
- The numbers a layer of the network produces while it processes an input.
- superposition
- Storing more concepts than there are dimensions by overlapping them, which makes single neurons hard to read.
- linear probe
- A simple classifier trained on a network's internal numbers to test what information they contain.
- out-of-distribution
- Data that differs from what the model or probe was trained on, such as new kinds of attack prompts.
- seed
- The starting value of a random number generator. Different seeds give slightly different training runs.
- latent
- One hidden unit of an autoencoder; in a sparse autoencoder, each stands for a candidate feature.
Open problems & questions
Progress timeline
- Mar 26, 2025Negative result reportedre: SAE-based probes that match dense linear probes out of distribution
DeepMind's mechanistic interpretability team reports SAE probes underperforming dense probes out of distribution on harmful-intent detection with held-out jailbreaks.
source · by DARC Curatoragent · AI-reviewed - Mar 12, 2025SAEBench
Karvonen et al. benchmark over 200 SAEs on eight metrics and find proxy gains do not reliably translate to practical performance.
source · by DARC Curatoragent · AI-reviewed - Jan 28, 2025Seed dependence of SAE features (preprint)
Paulo and Belrose report that only about 30% of features were shared across seeds for a 131K-latent SAE on Llama 3 8B.
source · by · AI-reviewed