Ambitious Mechinterp and Activation Geometry — reading collection
Iliad Intensive · Thursday 29 October 2026
Readings to accompany Eric Michaud and Brianna Grado-White’s lectures. This is more than a day’s reading, so skip freely, but the collection is a guided path in three parts: Anthropic’s Transformer Circuits programme and sparse autoencoders; then the methods that followed (transcoders, circuit tracing, feature geometry) alongside the middle-era work on testing circuit claims; then a miscellany of alternative approaches. Within each part, sections and readings are in suggested order. Importance scores prioritise this day’s reading: 5 marks the essential readings; 4 a recommended next choice; 3 useful depth; 2 an optional extension; 1 a specialist rabbit hole. Length is scored separately. You are not expected to read every high-rated item.
Start with Charles’s “What if not Circuits?”, section I. The rest of that essay is optional. As you read the papers below, ask what each takes a feature or circuit to be, what evidence would establish its existence, and what its methods actually measure.
Start here: what are we trying to explain?
What if not Circuits?, Charles Renshaw-Whitman, 2026
First reading: section I, “What Might We Mean By Circuits?” Distinguish assumptions about circuits from assumptions about representations; treat the essay as a perspective to examine, not a conclusion required by this notebook.
A Pragmatic Vision for Interpretability, Neel Nanda et al., 2025
The explicit turn toward task-grounded, empirically evaluated interpretability; compare its success criteria with the ambition of reverse-engineering a model.
Dmitry’s Koan, Dmitry Vaintrob, 2025
An explanation needs a specified precision scale: what error matters, relative to which baseline? Carry this question into the surrogate-model and causal-testing readings.
Part I — Transformer Circuits and sparse autoencoders
The core programme: superposition as the reason neurons are hard to read, dictionary learning as the proposed fix, and the evidence that made the field step back. Read the two anchors (Toy Models, Scaling Monosemanticity) before moving on.
Representations and Toy Models of Superposition
Toy Models of Superposition, Elhage et al., 2022
The synthetic setting where features are known: investigate when sparse features share dimensions, how interference is filtered, and which conclusions depend on the toy setup.
Distributed Representations: Composition & Superposition, Chris Olah, 2023
Distinguishes compositional codes, superposition, and mere changes of basis. This is the likely intended “representation types” note, and is also linked from Charles’s opening essay.
Exercises: interference and what a probe proves
In the tied-weight toy model, x\in\mathbb R^n, W\in\mathbb R^{m\times n}, m<n: h=Wx,\qquad \hat x=\operatorname{ReLU}(W^T h+b),\qquad \mathcal L=\mathbb E_x\sum_i I_i(x_i-\hat x_i)^2. Here I_i are feature-importance weights. These are synthetic features supplied by the experimenter.
- Writing w_i for column i of W, expand the preactivation of reconstructed feature i. Identify the self-term and interference. Why does sparsity help without making interference vanish?
- Encode each colour–shape pair with a separate one-hot coordinate. Construct a perfect linear probe for “red”. Does this prove that a dedicated red coordinate is used by the model’s computation?
Solution
- \hat x_i=\operatorname{ReLU}(\|w_i\|^2x_i+\sum_{j\ne i}w_i^Tw_jx_j+b_i). Sparsity reduces typical coactivation; it does not force the cross-inner-products to zero. Bias and ReLU can reject weak interference, at the cost of discarding some signal. A claim about performance also needs the distribution of feature magnitudes and co-occurrences.
- Assign probe weight 1 to every red–shape coordinate and 0 elsewhere. It predicts red perfectly on the one-hot inputs, although the code has no separate red coordinate. Decodability alone neither identifies the model’s chosen factorisation nor proves downstream causal use.
From dictionary learning to scaled features
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Bricken et al., 2023
The bridge from toy-model features to learned dictionaries in a small language model.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Templeton et al., 2024
Read the extraction method, examples and limitations together; feature activation, feature naming and steering provide different kinds of evidence.
Optional: SAE variants (Gated, Top-K, JumpReLU, Matryoshka)
Improving Dictionary Learning with Gated Sparse Autoencoders, Rajamanoharan et al., 2024
Separates feature selection from magnitude estimation to reduce shrinkage.
Scaling and evaluating sparse autoencoders, Gao et al., 2024
Top-K sparsity, scaling and evaluation; compare reconstruction with the separate tests of feature quality.
Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders, Rajamanoharan et al., 2024
Thresholded activations and an L0 objective, trained with straight-through estimators.
Learning Multi-Level Features with Matryoshka Sparse Autoencoders, Bussmann et al., 2025
Nested dictionaries as an attempt to capture features at multiple levels of abstraction.
Exercises: what the sparsity penalty buys
A schematic vanilla SAE has z=\operatorname{ReLU}(W_e(x-b_d)+b_e), \hat x=W_dz+b_d, and objective \mathbb E[\|x-\hat x\|_2^2+\lambda\|z\|_1], with decoder-column scale controlled. This schematic is not the objective of every SAE variant.
- For a unit decoder direction d and a fixed residual r, minimise \|r-ad\|^2+\lambda a over a\ge0. What bias does L1 introduce?
- Without controlling decoder scale, replace d by cd and a by a/c. What happens to reconstruction and the penalty?
- Propose a comparison in which better reconstruction would not by itself establish better interpretability.
Solution
- Differentiating yields a^*=\max(0,d^Tr-\lambda/2). The active coefficient is shrunk relative to the nonnegative least-squares estimate. The factor of two changes if reconstruction loss includes a factor 1/2.
- Reconstruction is unchanged while the penalty decreases by 1/c: an uncontrolled scaling degeneracy.
- Compare methods at matched sparsity and dictionary size, then separately measure held-out concept detection, intervention specificity and downstream loss recovery. A dictionary can improve reconstruction by encoding nuisance variation without improving the chosen task. None of these measurements alone certifies a complete ontology.
The SAE reckoning: negative results and changes of direction
The result and the argument that pushed groups away from SAEs as a general-purpose tool; together with the Pragmatic Vision, the motivation for Part II and much of Part III.
Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research, Lewis Smith et al., 2025
The GDM report linked from Charles’s essay: SAE-based approaches underperformed linear probes on the studied harmful-intent OOD detection task. Read the task and baselines before generalising the conclusion.
Activation space interpretability may be doomed, Chughtai and Bushnaq, 2025
An argument distinguishing structure visible in activations from mechanisms used in computation; a bridge to parameter decomposition, not a theorem that SAEs never help.
Part II — Transcoders, circuit tracing, feature geometry and the middle era
Two responses to the limits of Part I: replace parts of the model with sparse surrogates and trace circuits through them, and take seriously that features may lie on manifolds rather than along single directions. The final section steps back to the middle era of circuit discovery and causal testing, whose tools are what you need to judge any circuit claim.
Learned surrogates: transcoders and circuit tracing
Here the original model already exists. We train a more interpretable approximation to some of its computation, then test how well explanations of that approximation transfer to the original. Read the dense-to-sparse bridges experiment in the weight-sparse-transformer paper (Part III) as another example.
Transcoders Find Interpretable LLM Feature Circuits, Dunefsky, Chlenski and Nanda, 2024
A precursor to cross-layer transcoders: fit a wide, sparsely activating replacement to a trained dense MLP’s input–output map, then analyse the replacement’s feature interactions.
Circuit Tracing: Revealing Computational Graphs in Language Models, Ameisen et al., 2025
Cross-layer transcoders and attribution graphs: read the replacement-model construction and its limitations before interpreting edges as explanations of the original model.
On the Biology of a Large Language Model, Lindsey et al., 2025
Choose one case study, inspect its graph and interventions, and separate observations from the proposed computational story. Pair with the Circuit Tracing methods paper.
Exercise: reconstruction is not mechanism equivalence
Construct f,g:\mathbb R\to\mathbb R that agree on a training distribution supported on [-1,1] but disagree after an intervention that sets x=2. What additional checks would you want before transferring an attribution graph from a replacement model to the original?
Solution
Take f(x)=x and g(x)=x+\operatorname{ReLU}(x-1). They agree everywhere on the stipulated support, but f(2)=2 and g(2)=3. Check the original and replacement under the actual interventions, relevant held-out distributions, and matched behavioural metrics; inspect reconstruction-error contributions and mechanisms the replacement leaves fixed. Agreement on ordinary inputs is insufficient to guarantee agreement under interventions.
Feature manifolds and geometry (Goodfire)
If a concept is a curved manifold rather than a direction, a linear dictionary spends many latents tiling it. Start with Eric Michaud’s paper.
Understanding sparse autoencoder scaling in the presence of feature manifolds, Eric J. Michaud, Liv Gorton and Tom McGrath, 2025
Especially relevant to the day’s lecturer: dictionary capacity can be spent covering manifolds rather than identifying additional underlying features.
Can SAEs Capture Neural Geometry?, Bhalla, Fel et al., Goodfire, 2026
How linear dictionary elements can capture, fragment or dilute curved structure; compare the synthetic demonstrations with model examples.
Uncovering Neural Geometry in Vision Models With Block-Sparse Featurizers, Fel, Kowal et al., 2026
A move from individual directions to multidimensional subspaces; the empirical setting here is vision models.
Neural Networks Are Already Clean, Nick Cammarata, 2026
Optional contemporary counterpoint: an informal working hypothesis about neuron interpretability, not an established refutation of superposition.
Exercise: counting latents on a manifold
Let h(\theta)=(\cos\theta,\sin\theta). Distinguish its manifold dimension, linear-span dimension and the number of dictionary directions one might use to approximate it. Why might more SAE latents fail to imply more underlying concepts?
Solution
The circle is a one-dimensional manifold with a two-dimensional linear span. A dictionary can use arbitrarily many local directions to describe it. Counting those elements counts a chosen discretisation, not necessarily independent semantic factors.
The middle era: circuit discovery, causal scrubbing and causal abstraction
Causal Scrubbing: a method for rigorously testing interpretability hypotheses, Chan et al., Redwood Research, 2022
Read sections 2–4 and limitations: test a specified explanation with constrained resampling. Distinguish the original proposal from the later retrospective comments.
Towards Automated Circuit Discovery for Mechanistic Interpretability, Conmy et al., 2023
ACDC searches the original model’s computational graph for task-relevant edges using patching. Unlike a transcoder, it does not train a replacement MLP; dataset, metric and corruption scheme define the discovery problem.
How to use and interpret activation patching, Heimersheim and Nanda, 2024
Practical companion on corruption, noising versus denoising, redundancy and metric choice.
Causal Abstractions of Neural Networks, Geiger et al., 2021
Interchange interventions test whether a neural representation plays the causal role assigned by a higher-level model.
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations, Geiger et al., 2023; CLeaR 2024
Distributed Alignment Search learns subspaces aligned with hypothesised variables; distinguish fitting the alignment from testing it.
Causal Abstraction for Mechanistic Interpretability, Geiger et al., JMLR 2025
Longer theoretical reference: maps between high- and low-level interventions, unifying several interpretability methods. Read selected sections.
How useful is mechanistic interpretability? — Nanda / Redwood dialogue, Neel Nanda, Ryan Greenblatt, Buck Shlegeris and Oliver Habryka, December 2023
The Nanda/Redwood conversation: what induction-head and feature explanations actually explain, what would count as a faithful mechanism, and how to measure usefulness. Read it as a snapshot of the era’s open questions, then compare with the later Pragmatic Vision reading at the top.
Exercises: interchange and redundancy
For an orthogonal rotation R, coordinate projector P, base state h_b and source state h_s, a subspace interchange is h'=R^T[(I-P)Rh_b+PRh_s]=h_b+Q(h_s-h_b),\qquad Q=R^TPR.
- Show that Q is an orthogonal projector and that (I-Q)h'=(I-Q)h_b. What would an evaluation on the same pairs used to learn R fail to establish?
- Let clean inputs be A=B=1 and corrupted inputs A=B=0. Compare individually restoring one input and individually corrupting one input for C=A\land B and C=A\lor B.
Solution
- Q^T=Q and Q^2=R^TPRR^TPR=Q. Hence (I-Q)Q=0, proving complement preservation. Fitting an alignment to examples does not establish generalisation to held-out interventions, or rule out competing high-level explanations.
- For AND, restoring either input alone leaves 0; corrupting either alone destroys the clean output. For OR, restoring either alone gives 1; corrupting either alone leaves 1. The apparent importance of a component depends on intervention direction and redundancy.
Part III — Alternative approaches
A miscellany of departures from activation-space dictionaries: train interpretable models from scratch, decompose parameters instead of activations, read features off the empirical NTK, study activation plateaus, and build theory of computation in superposition. Sample according to taste.
Designed for interpretability: training networks from scratch
Here interpretability changes the architecture or constraints of the task model during its own training. Evaluate both the resulting explanations and the capability cost. These are different interventions from fitting an explanatory model to an already-trained network.
Weight-sparse transformers have interpretable circuits, Gao et al., 2025; ICML 2026
Start with the de novo models: constrain weights during training, then prune circuits for selected tasks. Examine the capability–interpretability tradeoff and the limits of scaling sparse models; the later dense-model bridge experiment is a separate surrogate approach.
Bilinear MLPs enable weight-based mechanistic interpretability, Pearce et al., 2024; ICLR 2025
Bilinear layers make weight-based analysis tractable in trained image and small language models. Exact quadratic structure helps analysis, but does not by itself assign human meanings to components.
B-cos Networks: Alignment is All We Need for Interpretability, Böhle, Fritz and Schiele, 2022
An optional vision example: train with transforms that encourage weight–input alignment and yield input-dependent contribution maps. This explanation target differs from recovering an LLM algorithm.
Exercise: tractable computation versus meaningful components
For a bias-free bilinear layer y=W_o[(W_ax)\odot(W_bx)], let a_j^T,b_j^T be the rows of W_a,W_b. Show that y_k=x^TQ_kx,\qquad Q_k=\frac12\sum_j(W_o)_{kj}(a_jb_j^T+b_ja_j^T). What does this exact representation buy, and what remains to be shown before calling its eigenvectors interpretable features?
Solution
Expand y_k=\sum_j(W_o)_{kj}(a_j^Tx)(b_j^Tx) and use x^Ta_jb_j^Tx=x^Tb_ja_j^Tx. Symmetrising gives the stated Q_k. Its eigendecomposition exposes exact quadratic contributions to this output coordinate. It does not establish semantic meaning, uniqueness within degenerate eigenspaces, or the role of those contributions in the complete network; these require further evidence.
Parameter decomposition
Decompose the weights rather than the activations, seeking parameter components that are simple, sparsely used and jointly faithful. The motivating argument, “Activation space interpretability may be doomed”, is in Part I.
Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition, Braun et al., 2025
APD seeks simple, sparsely used parameter components with faithful behaviour, demonstrated in toy systems.
Towards Scalable Parameter Decomposition, Bushnaq et al., Goodfire, 2025
SPD replaces attribution-based selection with stochastic ablation; a bridge from APD to subsequent methods.
Interpreting Language Model Parameters, Bushnaq et al., Goodfire, 2026
VPD and a 67M-parameter language model. Prioritise the introduction, section 2 and section 7, including “Attribution graphs are not computational graphs” and the adversarial robustness limitations.
Exercise: the choice of parts
For f(x)=W_2W_1x and invertible R, show that \widetilde W_1=RW_1, \widetilde W_2=W_2R^{-1} preserves the function. Does deleting hidden coordinate i before and after this change of basis generally preserve the same ablated function?
Solution
W_2R^{-1}RW_1=W_2W_1. Deleting coordinate i in the new basis instead inserts I-R^{-1}e_ie_i^TR between W_2 and W_1, generally different from I-e_ie_i^T. Exact functional or parameter reconstruction does not uniquely pick the parts whose interventions should be meaningful.
Feature discovery through the empirical NTK
Feature Identification via the Empirical NTK, Jennifer Lin, 2025
Eigenspaces of trained-model eNTKs recover known features in Toy Models of Superposition and a modular-addition MLP. Read the kernel variants and the rotation within feature subspaces; this is evidence in small models, not a general feature-identification theorem.
Mean field sequence: an introduction, Dmitry Vaintrob and Lauren Greenspan, 2026
A complementary theory-led approach to interpretable structure, including modular addition. Distinguish adaptive mean-field limits from the frozen NTK limit; computing a trained model’s eNTK does not assume its training was lazy.
A tale of three theories: sparsity, frustration, and statistical field theory, Dmitry Vaintrob, 2026
Optional preliminary research write-up connecting sparse denoising, interference and frustration-induced structure across scales. A bridge from TMoS and computation in superposition to statistical-physics explanations.
Exercise: a kernel subspace is not a unique feature dictionary
For scalar model output f_\theta, at a fixed trained checkpoint define J_{a\mu}=\frac{\partial f_\theta(x_a)}{\partial\theta_\mu},\qquad K=JJ^T. For vector outputs, one can retain sample–output pairs as matrix indices, or analyse a selected output’s kernel. Specify the convention before comparing spectra.
- Show that K is positive semidefinite. If Ku=\lambda u, \|u\|=1 and \lambda>0, show that v=J^Tu/\sqrt\lambda is a unit eigenvector of J^TJ.
- If a top eigenvalue is repeated, can the spectrum alone select a unique basis of features? What additional evidence would help?
Solution
- z^TKz=\|J^Tz\|^2\ge0. Also \|v\|^2=u^TKu/\lambda=1, and J^TJv=\lambda v. Data-space modes and parameter-space modes are related, but live in different spaces.
- Any orthogonal rotation inside a degenerate eigenspace is still an eigenbasis. A spectral gap can identify a subspace without identifying its preferred axes. Ground-truth feature alignment in a toy system, an explicitly motivated basis-selection criterion, and held-out causal tests provide different additional evidence. A large eigenvalue alone is not a semantic label.
Activation plateaus: Heimersheim and collaborators
Activation Plateaus: Where and How They Emerge, Matthew Shinkle and Stefan Heimersheim, 2025
Start here: interpolating early activations can yield flat downstream responses separated by sharp transitions. GPT-2 Large experiments trace their gradual sharpening across layers to MLP sensitivity.
[Interim research report] Activation plateaus & sensitive directions in GPT2, Stefan Heimersheim and Jake Mendel, 2024
The original perturbation experiments and error-correction motivation. Read the September 2024 correction: confidence in the sensitive-direction comparison was reduced after identifying baseline effects.
Evidence for feature-specific error correction in LLMs, Francisco Ferreira da Silva and Stefan Heimersheim, 2026
Optional follow-up comparing contrastive candidate feature directions with mixtures of directions, connecting plateau geometry to the error-correction hypothesis.
Exercise: a plateau along which direction?
Let F be the remainder of a network after a chosen activation layer. Along a straight interpolation h(t)=(1-t)h_A+th_B, define y(t)=F(h(t)). The chain rule gives \frac{dy}{dt}=J_F(h(t))(h_B-h_A).
- Does a small derivative along this path imply a small Jacobian in every direction? Give a counterexample.
- For residual blocks h_{\ell+1}=h_\ell+g_\ell(h_\ell), derive the recurrence for v_\ell=dh_\ell/dt. Why can sensitivity sharpen across layers, and why is multiplying the norms of the MLP Jacobians not the exact calculation?
- What comparison would help distinguish a learned plateau around real activations from an artefact of the chosen perturbation distribution?
Solution
- No. For F(x_1,x_2)=100x_2, a path parallel to the first coordinate has zero derivative although the Jacobian norm is 100. Flatness along one path need not imply a robust neighbourhood in all directions.
- v_{\ell+1}=[I+J_{g_\ell}(h_\ell)]v_\ell. Ordered Jacobian products can amplify some path segments more than others; sharpening is possible, not guaranteed. Residual identities and the alignment of vectors and singular directions matter. Products of operator norms give bounds, not generally equalities.
- Compare real and matched synthetic base activations with the same perturbation procedure; control scale, covariance, layer, path and output metric. Examine several directions and both norm-preserving and straight paths. Even a robust difference does not by itself identify discrete features or establish error correction as the unique explanation.
Computation in superposition
Mathematical Models of Computation in Superposition, Hänni, Mendel, Vaintrob and Chan, 2024
Constructive results for sparse computation in narrower networks; distinguish existence from what SGD learns in language models.
Compressed Computation is (probably) not Computation in Superposition, Bhagat et al., including Heimersheim, 2026
A particular toy benchmark’s apparent advantage is explained by a label-mixing term. This is a benchmark critique, not an impossibility theorem.
Compressed Computation under L⁴ Loss is likely Computation in Superposition, Ferreira da Silva and Heimersheim, 2026
Read beside the preceding negative result: a changed loss elicits behaviour explained using sparse codewords and pseudoinverse decoding.
Exercise: changing which errors matter
Compare residual vectors (2,0,0,0) and (1,1,1,1) under \sum_i e_i^2 and \sum_i e_i^4. Compute both gradients. Why is a change in training outcome under the new objective insufficient on its own to establish computation in superposition?
Solution
Both squared losses are 4; the fourth-power losses are 16 and 4. The coordinate gradients are 2e_i and 4e_i^3. Fourth-power loss penalises concentrated large errors more strongly. That motivates a different solution but does not identify its mechanism: inspect the learned representation, decoding and intervention evidence.