SAE-based probes that match dense linear probes out of distribution
Difficulty 45: the evidence conflicts by task (worse out of distribution on harmful intent and on average in four hard probing regimes, better on adversarial refusal samples), and retraining on chat data closed only half the gap. Application value 35: interpretable monitoring probes. Leverage 33: bears directly on the practical case for SAEs.
The problem
Can a probe reading at most 20 sparse-autoencoder features match a plain probe on unfamiliar data? Evidence conflicts.
- 1Dense probe reads all activations
- 2SAE probe reads ≤ 20 features
- 3Worse on new jailbreaks, better on refusal
- 4Settle on five fixed datasets
Probes. A probe is a simple classifier trained on a model's internal activations to read out a property, for example whether a prompt has harmful intent. A dense linear probe takes all the activation numbers (thousands of them), computes a weighted sum , and says "harmful" if it is positive. It is just a dot product, fitted to labelled examples. A probe built on a sparse autoencoder (SAE) instead uses at most of the SAE's features, here . Its appeal is interpretability: a person can read the 20 features and see why the probe fires.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- The transformer architectureAlgorithm
- Step 3
- Sparse autoencoders for interpretabilityAlgorithm · host
- Then
- SAE-based probes that match dense linear probes out of distributionThe problem
State of the art
The blog post reports dense linear probes near-perfect in and out of distribution (AUROC about 0.999 on held-out jailbreaks), while k-sparse SAE probes were noticeably worse out of distribution, and fine-tuning the SAE on chat data closed only about half of the gap. Kantamneni et al. (arXiv:2502.16681) find SAE probes underperform logistic regression on average in four hard regimes (data scarcity, class imbalance, label noise, covariate shift). In the other direction, Yeo et al. (arXiv:2505.23556, Findings of EMNLP 2025) report that probes on SAE refusal features generalise better than dense probes to adversarial out-of-distribution samples (WildJailbreak) for both Gemma-2-2B and Llama-3.1-8B. The evidence therefore conflicts across tasks and setups, which is why a fixed, multi-dataset comparison is needed.
Realistic applications
The practical case for SAEs over plain probes rests on interpretability of the readout; if sparse probes cannot match dense ones out of distribution, monitoring built on SAE features pays a robustness price.
What counts as solved
A public evaluation with code on the stated datasets, comparing k-sparse SAE probes with k at most 20 against dense linear probes on the same activations, with at least 3 SAE seeds and a result within the 1-point margin on the out-of-distribution split.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Datasets: at least five public text-classification sets with a distribution shift split (for example jailbreak versus ordinary prompts for harmful-intent detection), splits fixed before training. Metric: out-of-distribution accuracy (or AUROC) of k-sparse SAE probes with k at most 20, versus an L2-regularised dense logistic-regression probe on the same layer. Threshold: within 1 percentage point, averaged over datasets, and no dataset worse than 3 points. Budget: probe training is cheap; SAE training on chat data is a single-GPU job per SAE. Seeds: at least 3 SAE seeds.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Mostly evaluation and iteration on SAE training data and probe design.
Bottlenecks
ranked by votes- #1TheoryInformation lost in sparse reconstruction
Reconstruction error may drop exactly the directions a dense probe uses.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to Seed-stable sparse autoencoder dictionaries: 80% shared features at 131K latents
Progress
- Mar 26, 2025Negative result reportedsource
DeepMind's mechanistic interpretability team reports SAE probes underperforming dense probes out of distribution on harmful-intent detection with held-out jailbreaks.