Seed-stable sparse autoencoder dictionaries: 80% shared features at 131K latents
Difficulty 55: no method approaches 80% overlap on language models, and the low overlap may reflect a genuine non-uniqueness of sparse decompositions, which would make the target unreachable. Application value 26: indirect, through trust in dictionaries. Leverage 39: claims built on individual SAE features depend on it.
The problem
Can sparse autoencoders be trained so 80% of features reappear when retrained with a different seed?
- 1Train SAE twice, change seed
- 2Only about 30% features match
- 3Design more stable training
- 4Reach 80% without losing quality
Background. A sparse autoencoder (SAE) is a tool for reading what a language model represents internally. At some layer the model produces a vector of a few thousand numbers per token, its activations. The SAE learns a large dictionary of directions in that space, here 131,072 of them, and rebuilds each activation vector as a sum of only a few dictionary directions, , where most coefficients are zero. Each direction is called a feature, and the hope is that each one means one human-readable thing, such as "French text" or "a closing bracket".
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- The transformer architectureAlgorithm
- Step 3
- Sparse autoencoders for interpretabilityAlgorithm · host
- Then
- Seed-stable sparse autoencoder dictionaries: 80% shared features at 131K latentsThe problem
State of the art
The preprint states that seed-dependence appeared across multiple layers of three language models, two datasets and several SAE architectures, and that ReLU SAEs with L1 loss were more stable across seeds than TopK SAEs. Whether the low overlap reflects an unstable optimisation or genuinely many equivalent decompositions of the activations is not settled.
Realistic applications
If features are an artefact of the seed, claims built on a single dictionary (including circuit analyses and steering vectors) need replication across seeds; a stable method would make dictionaries comparable between labs.
What counts as solved
A public method and checkpoints for at least three seeds, with the shared-feature fraction computed by released code under the matching criterion of Paulo and Belrose, at least 80% for all seed pairs, and SAEBench scores within the baseline's variance.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Model and layer: Llama 3 8B at the layer used by the source paper, 131K latents, same training tokens as the baseline. Metric: fraction of latents with a match in the other seed's SAE under the source paper's criterion (released code), computed over all pairs of at least 3 seeds. Threshold: 80% or more for every pair, at the baseline's mean squared reconstruction error and L0, and SAEBench scores not worse than the baseline's seed-to-seed spread. Budget: three SAE trainings at 131K width on the baseline's token count (a single-GPU-days to GPU-weeks scale per SAE depending on tokens).
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Needs a new training idea, then compute for SAEs at 131K width.
Bottlenecks
ranked by votes- #1TheoryNon-uniqueness of sparse decompositions
Many sparse dictionaries may reconstruct activations equally well, so consistency may conflict with other objectives.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to SAE-based probes that match dense linear probes out of distribution
Progress
- Jan 28, 2025Seed dependence reported (preprint)source
About 30% of features shared across seeds at 131K latents on Llama 3 8B.