Beating the best circuit-localization method on the MIB benchmark
Difficulty 32: an incremental leaderboard improvement on a fixed benchmark, easier than the SAE questions (45 to 55). Application value 24: indirect, through better circuit tools. Leverage 19: the benchmark covers small models, so a record says little about frontier-scale analysis.
The problem
Which method finds the parts of a model that produce a behaviour best on the shared MIB test?
- 1Pick task and model
- 2Method selects components
- 3Score faithfulness at fixed size
- 4Beat the leaderboard
The idea of a circuit. A language model is built from components: attention heads, which let a token gather information from earlier tokens, and feed-forward blocks. For a narrow behaviour, such as completing "When Mary and John went to the store, John gave a drink to ___" with "Mary", only a small subset of components and the connections between them may matter. That subset is called a circuit. Finding it is like tracing which wires in a large electronic board carry a particular signal.
How circuits are found. The basic tool is activation patching: run the model on a clean input and on a slightly altered one, swap one component's output between the runs, and see how much the answer changes. A component whose swap changes the answer a lot is part of the circuit. Patching every component one by one is too slow for large models, so methods approximate it, for example by using derivatives to estimate each swap's effect in one pass (attribution), or by learning a mask that switches components on or off.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- The transformer architectureAlgorithm
- Step 3
- Mechanistic circuits and causal interventionsTheory · host
- Then
- Beating the best circuit-localization method on the MIB benchmarkThe problem
State of the art
The MIB paper reports that attribution and mask optimisation methods perform best on circuit localization, and that SAE features are not better than neurons on the causal-variable localization track. The causal-variable track, not this one, is where SAE features did no better than neurons. The public leaderboard is a Hugging Face space (mib-bench/leaderboard), live on 2026-10-17; no figures are quoted here because leaderboard values change and were not independently rerun.
Realistic applications
A better localization method means circuits that explain more of a model's behaviour with fewer components, which is the prerequisite for scaling circuit-level explanations beyond hand analysis of small models.
What counts as solved
A submission to the MIB circuit-localization leaderboard with higher scores than the best prior entry on the benchmark's aggregate metric, reproduced from released code.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: MIB circuit-localization track, all four tasks and five models as defined in the paper, evaluated with the official code. Metric: the benchmark's aggregate circuit faithfulness score at matched circuit sizes. Threshold: strictly above the best entry on the official leaderboard at submission time. Budget: the benchmark's published compute requirements; report seeds for any stochastic method.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Method design evaluated on a fixed benchmark.
Bottlenecks
ranked by votes- #1TheoryFaithfulness without exhaustive patching
Exact edge importance needs many forward passes; cheap approximations have known errors.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.
Progress
- Apr 17, 2025MIB releasedsource
Benchmark with circuit-localization and causal-variable tracks introduced.