MDL4 — Mysteries of Deep Learning: Universalities and a Hundred Surprises

Mysteries of Deep Learning, Iliad Intensive 2026

Different networks trained on similar data end up with the same smooth loss curves, nearly regardless of architecture, and arguably with similar internal representations. Nobody designed these regularities in, and nobody fully knows why they hold: the machines, they just want to learn. Sections 1–4 are about those universalities. Sections 5–14 are a catalogue of further things trained networks do that no theory predicted, one per section with the papers that report them: read until you find something you didn’t know. Together they are a lower bound on what any good theory of deep learning must explain.

Universalities

The catalogue (sections are independent)

1. Keystone: scaling laws

Why it’s here: the loss of a language model is a predictable, smooth function of scale, and the details of the network barely enter. Both facts are empirical, and both are unexplained.

Scaling laws for neural language models, Kaplan et al. (2020) Importance ●●●●● Length ●●●○○

Test loss is a power law in parameters, data and compute, while width, depth and shape barely matter at fixed parameter count. The abstract, Section 3 and Figures 1 and 5–7 carry the argument.

Smooth power laws nobody designed in. Kaplan et al. train transformer language models with up to about 10^9 non-embedding parameters N. When not bottlenecked by the others, test loss falls as L(N)\approx(N_c/N)^{\alpha_N} with \alpha_N\approx0.076, and as similar power laws in dataset size D (\alpha_D\approx0.095) and in compute (\alpha_C\approx0.050), with some trends spanning more than seven orders of magnitude. Loss on other text datasets follows the same power with an offset. Nothing in the architecture, the optimizer or the data was chosen to produce this. It made training a much larger model a predictable investment rather than a gamble. Fitting these curves and allocating compute between parameters and data (Chinchilla) is in MDL2, section ‘Why scale? Should you consume more tokens?’.

Architecture mostly sets constants. At fixed N, the loss varies only a few percent over a wide range of shapes. The aspect ratio d_{\rm model}/n_{\rm layer} can vary by a factor of 40 at little cost (Figure 5), and models of different depths fall on one curve once embedding parameters are excluded (Figure 6). §3 extends this across architecture families: most changes shift the curve by a constant factor in compute and rarely change its slope. Kaplan’s exception is instructive. LSTMs match transformers on early tokens of the context but cannot use long contexts, so they scale worse (Figure 7). This is another bitter lesson: what matters is scale, not cleverness in the layers.

2. Why power laws? A hundred unnecessary explanations

Very different mechanisms produce the same power-law learning curve, so the curve’s form alone cannot tell you which one is at work.

The quantization model of neural scaling, Michaud, Liu, Girit & Tegmark (NeurIPS 2023) Importance ●●●●● Length ●●●○○

Networks learn discrete skills (‘quanta’) in order of use frequency; Zipfian frequencies give power-law scaling. Decomposes model behaviour into candidate quanta using gradients.

Supplemental

On the asymptotic learning curves of kernel ridge regression under power-law decay, Li, Zhang & Lin (2023) Importance ●○○○○ Length ●●●●○

Rigorous kernel-ridge learning curves for power-law spectra, including the saturation of Exercise 2.2. A rabbit hole.

Exercises: power-law learning curves

Two toy models with the same answer; the second assumes Zipf’s law for how often skills are used.

Exercise 2.1. Two routes to one curve.

Spectral model. Features x\in\mathbb R^\infty are Gaussian with covariance \mathrm{diag}(\lambda_k), \lambda_k=k^{-\alpha}, \alpha>1. The target is y=\sum_k\beta_kx_k+\varepsilon, with noise variance \sigma^2, and we have n samples. The excess loss of an estimate \hat\beta is \sum_k\lambda_k(\hat\beta_k-\beta_k)^2. The signal power in mode k is c_k^2=\lambda_k\beta_k^2=k^{-\alpha'} with \alpha'>1 (a generic target has \alpha'=\alpha; smoother targets have \alpha'>\alpha).

Quanta model. A task consists of discrete skills k=1,2,\dots, used with frequencies p_k\propto k^{-(1+a)}, a>0. A model that has learned the N most frequent skills has loss proportional to the frequency mass of the rest.

(a) Show that \sum_{k>K}k^{-s}\sim K^{1-s}/(s-1) for s>1.

(b) Heuristic for ridge. Suppose the estimator learns modes 1,\dots,K with K\le n and nothing beyond. Each learned mode costs about \sigma^2/n (error variance \sigma^2/(n\lambda_k), weighted by \lambda_k); each unlearned mode costs c_k^2. So E(K)\approx\sum_{k>K}c_k^2+\frac{\sigma^2K}n . Show that E\sim n^{-(\alpha'-1)} without noise, and E\sim n^{-(\alpha'-1)/\alpha'} with noise and the best K.

(c) Show that the quanta model’s loss after learning N skills scales as N^{-a}. If N grows as a power of parameters or data, what form does the loss curve take?

Solution

(a) k^{-s} is decreasing, so \int_{K+1}^\infty t^{-s}dt\le\sum_{k>K}k^{-s}\le\int_K^\infty t^{-s}dt. Both integrals are \sim K^{1-s}/(s-1).

(b) With s=\alpha', the tail is \sim K^{-(\alpha'-1)}/(\alpha'-1). Without noise, take K=n: E\sim n^{-(\alpha'-1)}. With noise, dE/dK=0 gives -K^{-\alpha'}+\sigma^2/n=0, so K^*=(n/\sigma^2)^{1/\alpha'}, below n for large n. Both terms are then of order (K^*)^{1-\alpha'}, so E\sim(n/\sigma^2)^{-(\alpha'-1)/\alpha'} .

(c) By (a) with s=1+a, \sum_{k>N}p_k\propto N^{-a}/a. If N\propto P^\mu for a resource P, the loss is \propto P^{-a\mu}, again a power law, with nothing spectral used. An observed power law cannot by itself tell the two mechanisms apart. Finer measurements can: per-example loss curves should be step-like under quanta and smooth under the spectral picture.

Exercise 2.2. Where the heuristic fails: saturation.

Use the spectral model of Exercise 2.1. Ridge shrinks mode k by the filter \lambda_k/(\lambda_k+\kappa), so it also biases the modes it “learns”. At large n the effective regularization \kappa sits near the eigenvalue of the last resolved mode; without noise take \kappa\approx\lambda_n=n^{-\alpha}. The bias from shrinking the head is B_{\rm head}=\sum_{k<n}c_k^2\Big(\frac{\kappa}{\lambda_k+\kappa}\Big)^2 .

(a) Show that B_{\rm head}\approx n^{-2\alpha}\sum_{k<n}k^{2\alpha-\alpha'} up to constants, and that modes with k comparable to n contribute at most the order of the tail, n^{1-\alpha'}.

(b) Evaluate the sum when \alpha'-1<2\alpha, when \alpha'-1>2\alpha, and at \alpha'-1=2\alpha. Conclude that the noiseless rate is n^{-\min(\alpha'-1,\,2\alpha)} away from the boundary, and find the rate on it.

(c) With noise and optimally tuned ridge, redo Exercise 2.1(b) with s=\min(\alpha'-1,2\alpha) in place of \alpha'-1, taking the cutoff K as free. Show that E\sim n^{-s/(s+1)} away from the boundary.

Hint

(a) For k\le n/2, \lambda_k\ge2^\alpha\kappa, so \kappa/(\lambda_k+\kappa) is within a constant of \kappa/\lambda_k=n^{-\alpha}k^\alpha. For n/2<k<n the filter factor is at most 1. (b) \sum_{k<n}k^q grows like n^{q+1} for q>-1, like \log n for q=-1, and converges for q<-1.

Solution

(a) For k\le n/2, \big(\kappa/(\lambda_k+\kappa)\big)^2\asymp n^{-2\alpha}k^{2\alpha}, so the summand is \asymp n^{-2\alpha}k^{2\alpha-\alpha'}. For n/2<k<n the factor is at most 1, and those modes contribute at most \sum_{n/2<k<n}c_k^2\lesssim n^{1-\alpha'}, the order of the tail.

(b) Let q=2\alpha-\alpha'.

  • \alpha'-1<2\alpha (q>-1): the sum is \sim n^{q+1}, so B_{\rm head}\sim n^{-(\alpha'-1)}, the order of the tail. The heuristic’s exponent is right.
  • \alpha'-1>2\alpha (q<-1): the sum converges, so B_{\rm head}\sim n^{-2\alpha}, larger than the tail. Ridge saturates at n^{-2\alpha}, however smooth the target.
  • \alpha'-1=2\alpha (q=-1): the sum is \sim\log n, so B_{\rm head}\sim n^{-2\alpha}\log n.

(c) With cutoff K (\kappa\approx\lambda_K), the same argument gives bias \sim K^{-s} and variance \sim\sigma^2K/n. Minimizing gives K^*\sim(n/\sigma^2)^{1/(s+1)} and E\sim n^{-s/(s+1)}. For s=\alpha'-1 this is Exercise 2.1(b). On the boundary there is an extra factor (\log n)^{1/(s+1)}.

These are heuristic scalings, valid up to constants. None of it needs the model to learn a good representation: plain ridge suffices. In both toy models the exponent is set by the data (\alpha,\alpha', or a), except where the learner’s own limits bind (ridge’s saturation at 2\alpha, or the quanta model’s acquisition rate \mu). That is §3’s theme.

▶ Interactive figure: spectral_filters.html. Panel B plots the deterministic-equivalent ridge risk (an asymptotic approximation) and Exercise 2.1’s heuristic against n, with Monte Carlo ridge points. Toggle noise, and push \alpha' past 2\alpha+1 to see saturation.

3. Another bitter lesson: architecture doesn’t matter (much)

Since 2017, challengers to and refinements of the transformer have mostly moved the scaling curve by a constant factor in compute, rarely changed its slope, and where slopes measurably differ the alternatives usually scale worse.

The bitter lesson, Sutton (2019) Importance ●●●●● Length ●○○○○

Seventy years of AI research: general methods that exploit more computation beat methods built on human knowledge of the domain, every time the computation arrives. Summary and reactions on Wikipedia.

Algorithmic progress in language models, Ho et al. (Epoch AI, 2024; paper) Importance ●●●●● Length ●●○○○

Estimates the transformer’s compute-equivalent gain at about 7\times (95% CI roughly 3–46\times), and required compute halving every 8 months or so. Treats each innovation as a constant multiplier, so cannot see slope changes.

Supplemental

Deep learning scaling is predictable, empirically, Hestness et al. (2017) Importance ●●●○○ Length ●●●○○

Power-law learning curves (against data, not compute) in translation, language, images and speech: “model improvements only shift the error but do not appear to affect the power-law exponent.”

Scaling laws vs model architectures: how does inductive bias influence scaling?, Tay et al. (2022) Importance ●●●○○ Length ●●●○○

The counterpoint: compute slopes for about ten architectures differ, with the vanilla transformer steepest, so architectures that win small can lose at scale. Few points per fit.

Neural scaling law, Wikipedia Importance ●●●○○ Length ●●○○○

Survey of the empirical scaling laws across modalities and of the proposed explanations, with the main papers linked.

Also: Mamba (Gu & Dao 2023; explainers: Grootendorst’s visual guide, The annotated S4), the strongest recent challenger, a selective state-space model whose scaling curve runs roughly parallel to a strong transformer recipe (their Figure 4); and I-JEPA (Assran et al. 2023), a different objective rather than different layers: predict the representation of a masked region, not its pixels.

In vision, NFNet CNNs match vision transformers at equal compute (MDL2, section ‘The opposite thesis: symmetry as inductive bias’).

Exercises: what a constant factor is worth

Exercise 3.1 (Fermi). Write the reducible loss as R(C)=A\,C^{-\alpha} in training compute C. Take \alpha\approx0.05 (Kaplan et al.’s exponent for loss against compute), and redo (a) with \alpha\approx0.15, roughly the Chinchilla fit’s exponent for the reducible loss at compute-optimal allocation (MDL2, section ‘Why scale? Should you consume more tokens?’).

(a) Architecture B has R_B(C)=(A/m)\,C^{-\alpha} with m>1: same exponent, better constant. Show that B at compute C matches A at compute m^{1/\alpha}C for every C. Ho et al. estimate the transformer’s gain at about 7\times in compute. What reduction of reducible loss is that? What about the roughly 10^4\times of training compute separating GPT-2 from frontier models a few years later?

(b) B instead has a slightly better exponent: R_B(C)=A\,C^{-(\alpha+\delta)}, with C in units of the compute where the curves cross. Show that B’s compute multiplier at C is C^{\delta/\alpha}. With \delta/\alpha=0.1 (a 10% better exponent), what is it three and ten orders of magnitude past the crossing?

(c) An architecture paper typically spans two or three decades of compute. Why does that make “a constant-factor gain” and “a better exponent” hard to tell apart, and which way would a small-scale advantage from a helpful inductive bias tend to mislead?

Solution

(a) R_A(kC)=A\,k^{-\alpha}C^{-\alpha} equals (A/m)C^{-\alpha} when k=m^{1/\alpha}, independent of C: on log–log axes B is A shifted left by \log m/\alpha. Conversely m=k^\alpha. With \alpha=0.05: 7^{0.05}\approx1.10, a 10% lower reducible loss, and 10^{4\times0.05}\approx1.58, 37% lower. With \alpha=0.15: 7^{0.15}\approx1.34 (25%) and 10^{0.6}\approx4 (75%). Because \alpha is small, a modest loss improvement is a large compute multiplier, and four orders of magnitude would need a far larger constant-factor improvement than any architecture change has delivered.

(b) A reaches A\,C^{-(\alpha+\delta)} at C_A=C^{1+\delta/\alpha}, so the multiplier is C^{\delta/\alpha}: 10^{0.3}\approx2 three decades past the crossing and 10 ten decades past it. A better exponent wins eventually, but slowly; over the explored range it looks like a constant factor.

(c) Over two or three decades a 10% exponent difference changes the multiplier by about 2\times, comparable to fitting noise and to tuning one architecture better than another, so the two hypotheses give nearly the same plot. An inductive bias that suits the data helps most when data or compute is scarce, so it shows up as a small-scale advantage that shrinks: the better-looking architecture has the shallower slope. This is Tay et al.’s pattern, and extrapolating such a win is the mistake the bitter lesson warns against.

4. Same representation? CKA, CCA, and the Platonic and linear representation hypotheses

Different networks seem to learn similar representations, perhaps converging on one shared one whose concepts are directions; each claim means something only once you say which transformations of a representation count as cosmetic.

The Platonic representation hypothesis, Huh, Cheung, Wang & Isola (ICML 2024) Importance ●●●●● Length ●●●○○

Representations across models and modalities align more as models get more capable (by mutual nearest neighbours); conjectures convergence to a shared statistical model of the world.

The linear representation hypothesis and the geometry of large language models, Park, Choe & Veitch (ICML 2024) Importance ●●●●● Length ●●●○○

Makes “concepts are directions” precise with counterfactual pairs, and argues the geometry needs a chosen ‘causal inner product’ (Exercise 4.2).

Supplemental

Similarity of neural network representations revisited, Kornblith, Norouzi, Lee & Hinton (ICML 2019) Importance ●●●●○ Length ●●●○○

Indices invariant to every invertible linear map (CCA included) go blind once width reaches the number of data points; proposes CKA, which matches corresponding layers across seeds. Background for Exercise 4.1.

Proof of a perfect Platonic representation hypothesis, Ziyin & Chuang (2025) Importance ●●●○○ Length ●●○○○

Deep linear networks of any depth and width, seeing different invertible views of the same noisy-label data, have every hidden layer aligned up to rotation and scale at the minima SGD’s gradient noise selects. Elementary proof; needs label noise, and does not show SGD gets there.

Not all language model features are one-dimensionally linear, Engels, Michaud, Liao, Gurnee & Tegmark (ICLR 2025) Importance ●●●○○ Length ●●●○○

The counterpoint: irreducibly multi-dimensional features, such as circles for days and months, used for modular arithmetic.

Mechanistic interpretability: linear representation hypothesis, Wikipedia Importance ●●○○○ Length ●○○○○

A short statement of the hypothesis and of superposition, placed in the field that relies on them.

SVCCA, model stitching and superposition are in the catalogue below (sections ‘Early training decides: bottom-up learning and critical periods’, ‘Model stitching works’ and ‘Superposition and adversarial attacks’).

Exercises: similarity invariances

Exercise 4.1. Invariance groups, and why a minimum does not fix the representation.

Stack a layer’s activations on n inputs from two networks as rows of X\in\mathbb R^{n\times p_1}, Y\in\mathbb R^{n\times p_2}, columns centered, with Gram matrices K=XX^\top, L=YY^\top. Linear CKA is \langle K,L\rangle_F/(\|K\|_F\|L\|_F). CCA’s canonical correlations are the cosines of the principal angles between the column spaces of X and Y.

(a) Show that \langle K,L\rangle_F=\|Y^\top X\|_F^2, and that CKA is unchanged by X\mapsto XQ with Q orthogonal and by X\mapsto cX, c\neq0. Show that it is changed by the stretch A=\mathrm{diag}(2,1) applied to the four centered points (\pm1,0),(0,\pm1).

(b) Show that CCA is unchanged by X\mapsto XA for every invertible A. If centered X and Y both have rank n-1, show that all canonical correlations are 1, whatever the two networks compute.

(c) Take a deep linear network f(x)=W_D\cdots W_1x and an invertible T. Show that W_i\to TW_i, W_{i+1}\to W_{i+1}T^{-1} leaves f, and hence the loss, unchanged, while the layer-i representation h becomes Th. If the centered representations span \mathbb R^{d_i}, show that \mathrm{CKA}(h,Th)=1 only if T is a scaled orthogonal matrix.

Hints

(a) Use cyclicity of the trace, and \|K\|_F=\|X^\top X\|_F. (b) CCA depends only on column spaces; every centered column is orthogonal to \mathbf 1. (c) CKA is 1 iff the Gram matrices are proportional: h^\top T^\top Th'=c\,h^\top h' on a spanning set.

Solution

(a) \langle K,L\rangle_F=\mathrm{Tr}(XX^\top YY^\top)=\mathrm{Tr}(Y^\top XX^\top Y)=\|Y^\top X\|_F^2. Under X\mapsto XQ, K\mapsto XQQ^\top X^\top=K. Under X\mapsto cX, K\mapsto c^2K, which cancels. For the stretch, with Y=X and X'=XA: X^\top X=2I, X^\top X'=\mathrm{diag}(4,2) with squared norm 20, \|X'^\top X'\|_F=\|\mathrm{diag}(8,2)\|_F=2\sqrt{17}, \|X^\top X\|_F=2\sqrt2. So \mathrm{CKA}=20/(4\sqrt{34})=5/\sqrt{34}\approx0.86. CKA’s group is orthogonal maps and isotropic scaling, not all of GL.

(b) \mathrm{col}(XA)=\mathrm{col}(X) for invertible A. Centered columns lie in \mathbf 1^\perp, of dimension n-1, so rank n-1 forces \mathrm{col}(X)=\mathbf 1^\perp=\mathrm{col}(Y) and all principal angles vanish. This is Kornblith et al.’s point: invariance to every invertible linear map is too much for wide layers.

(c) W_{i+1}T^{-1}TW_i=W_{i+1}W_i, and the layer-i output becomes Th. CKA is 1 iff h^\top T^\top Th'=c\,h^\top h' for all pairs; if the h span, bilinearity gives T^\top T=cI. For d_i\ge2, T=\mathrm{diag}(2,1,\dots,1) is not scaled orthogonal. So every global minimum of a deep linear network sits on a GL(d_i) orbit of global minima whose representations differ by CKA. Being a minimum fixes the function, not the representation; convergence of representations must come from whatever selects a point on the orbit, which is what Ziyin and Chuang analyse.

Exercise 4.2. Which inner product?

A language model’s next-token logits are \lambda(x)^\top\gamma(y), where \lambda(x)\in\mathbb R^d is the final hidden state for context x and \gamma(y) the unembedding of token y. Concept directions are differences such as \gamma(y_1)-\gamma(y_0). Park et al.’s causal inner product is \langle a,b\rangle_{\mathrm C}=a^\top\mathrm{Cov}(\gamma)^{-1}b, with the covariance over the vocabulary (assume it invertible).

(a) Show that for any invertible A, \lambda\to A\lambda, \gamma\to A^{-\top}\gamma leaves every logit, and so the model’s behaviour, unchanged.

(b) Give two unembedding differences that are Euclidean-orthogonal before such a change and not after.

(c) Show that \langle a,b\rangle_{\mathrm C}, recomputed in the new coordinates, equals its old value.

Hint

(c) If \gamma'=A^{-\top}\gamma, then \mathrm{Cov}(\gamma')=A^{-\top}\mathrm{Cov}(\gamma)A^{-1}.

Solution

(a) (A\lambda)^\top(A^{-\top}\gamma)=\lambda^\top A^\top A^{-\top}\gamma=\lambda^\top\gamma.

(b) In \mathbb R^2, a=e_1, b=e_2 and A^{-\top}=\begin{pmatrix}1&1\\0&1\end{pmatrix} give a'=(1,0), b'=(1,1), with a'\cdot b'=1. Euclidean angles between concept directions are not fixed by the model’s behaviour.

(c) a'=A^{-\top}a, b'=A^{-\top}b and \mathrm{Cov}(\gamma')^{-1}=A\,\mathrm{Cov}(\gamma)^{-1}A^\top, so a'^\top\mathrm{Cov}(\gamma')^{-1}b'=a^\top A^{-1}A\,\mathrm{Cov}(\gamma)^{-1}A^\top A^{-\top}b=a^\top\mathrm{Cov}(\gamma)^{-1}b .

Behaviour fixes the representation only up to GL(d), the same group as Exercise 4.1(c), so a geometric claim needs extra structure. Whitening by the vocabulary covariance supplies an invariant geometry; whether it is the right one is what Park et al. argue from their assumption about causally separable concepts.

5. Transformer middle layers

Intermediate layers of language models often give better general-purpose features than the final layer, even though the final layer is the one the training objective shapes most directly.

Layer by layer: uncovering hidden representations in language models, Skean, Arefin, Zhao, Patel, Naghiyev, LeCun & Shwartz-Ziv (ICML 2025) Importance ●●●●● Length ●●●○○

On 32 embedding tasks, intermediate layers of transformers and state-space models often beat the last layer; label-free measures of representation quality track how this varies with depth.

6. Early training decides: bottom-up learning and critical periods

Gradient descent updates every layer at once, yet lower layers converge first and upper layers later, and, as in animal vision, a deficit early in training (blurred images) leaves a permanent deficit while the same deficit later does little harm: what is decided early is not undone by later training.

SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability, Raghu, Gilmer, Yosinski & Sohl-Dickstein (NeurIPS 2017) Importance ●●●●● Length ●●●○○

Introduces SVCCA for comparing layer representations and finds low effective dimensionality and bottom-up convergence, which motivates freezing lower layers in sequence to save compute.

Supplemental

Critical learning periods in deep neural networks, Achille, Rovere & Soatto (ICLR 2019) Importance ●●●●○ Length ●●●○○

Early blur leaves a lasting deficit but early vertical flips do not; the Fisher information in the weights rises then falls during training, which the authors tie to the critical period.

7. Model stitching works

The bottom of one trained network joins the top of another through a single linear layer with little loss, so separately trained networks represent things almost the same way (see MDL4, section ‘Same representation? CKA, CCA, and the Platonic and linear representation hypotheses’).

Revisiting model stitching to compare neural representations, Bansal, Nakkiran & Barak (NeurIPS 2021) Importance ●●●●● Length ●●●○○

Networks from different seeds, or supervised versus self-supervised training, stitch with little loss; representations learned with more data, width or training improve weaker models when stitched in.

8. Superposition and adversarial attacks (and on you too)

Networks store more sparse features than they have dimensions, in almost-orthogonal directions; adversarial examples may exploit this interference, or features that are predictive but brittle, and some even bias briefly shown humans (see MDL4, section ‘Same representation? CKA, CCA, and the Platonic and linear representation hypotheses’).

Toy models of superposition, Elhage et al. (Anthropic, 2022; also arXiv 2209.10652) Importance ●●●●● Length ●●●●○

Toy ReLU autoencoders store sparse features in superposition, with a phase change in sparsity and importance and features arranged as regular polytopes; ‘Superposition as a Phase Change’ has the figure’s phase diagram.

Supplemental

Adversarial examples are not bugs, they are features, Ilyas, Santurkar, Tsipras, Engstrom, Tran & Madry (NeurIPS 2019) Importance ●●●○○ Length ●●●○○

Training only on adversarial images relabelled with the attack’s target class, which look mislabelled to humans, still gives non-trivial clean accuracy: the perturbations carry real, predictive, non-robust features.

Adversarial examples that fool both computer vision and time-limited humans, Elsayed, Shankar, Cheung, Papernot, Kurakin, Goodfellow & Sohl-Dickstein (NeurIPS 2018) Importance ●●○○○ Length ●●○○○

Perturbations built to transfer across an ensemble of models with a retina-like front end measurably bias people’s choices when images are shown for a fraction of a second.

Also: Adversarial machine learning on Wikipedia, for the kinds of attack and defence and the main papers.

▶ Interactive figure: superposition.html. Panel 1 is the two-feature, one-dimension phase diagram over density and relative importance (dashed: the closed form of Exercise 8.2; solid: optimized weights and biases; dots: trained models). Panel 2 trains n=5 or 6 features in m=2 dimensions in your browser: as sparsity grows, features go from dedicated dimensions to antipodal pairs to a regular polygon.

Exercises: packing directions and the two-feature toy

Exercise 8.1 (packing almost-orthogonal directions). Let u,v be independent, uniformly random unit vectors in \mathbb R^m.

  • (a) Show that \mathbb E\langle u,v\rangle=0 and \mathbb E\langle u,v\rangle^2=1/m.
  • (b) Take as given \Pr(|\langle u,v\rangle|>\varepsilon)\le2e^{-m\varepsilon^2/2}. Show that if N^2e^{-m\varepsilon^2/2}<1, then N independent random unit vectors have all pairwise |\cos|\le\varepsilon with positive probability. Conclude that \mathbb R^m contains about e^{m\varepsilon^2/4} such directions.
Hint

(a) Condition on v and use \mathbb E[uu^\top]=I/m (isotropy, and the trace is 1). (b) Union bound over pairs.

Solution

(a) u and -u have the same distribution, so \mathbb E\langle u,v\rangle=0. By isotropy \mathbb E[uu^\top]=\kappa I, and taking the trace gives \mathbb E\|u\|^2=1=\kappa m. Conditioning on v, \mathbb E\langle u,v\rangle^2=v^\top\mathbb E[uu^\top]v=1/m. A typical overlap is of size 1/\sqrt m.

(b) There are N(N-1)/2 pairs. By the union bound, \Pr(\text{some pair has }|\cos|>\varepsilon)\le\frac{N(N-1)}2\cdot2e^{-m\varepsilon^2/2}<N^2e^{-m\varepsilon^2/2}<1, so with positive probability no pair is bad, and such a configuration exists. The condition holds for any N<e^{m\varepsilon^2/4}: exponentially many directions in m, against only m exactly orthogonal ones. If features are sparse, the small overlaps rarely matter at the same time, which is why storing more than m of them can pay.

Exercise 8.2 (the smallest superposition). The toy model of Elhage et al. compresses features x\in\mathbb R^n to Wx\in\mathbb R^m and reads them back as \hat x=\mathrm{ReLU}(W^\top Wx+b), with loss \sum_iI_i\,\mathbb E(\hat x_i-x_i)^2 for importances I_i. Take n=2, m=1, importances 1 and r\le1, and features that are independently 0 with probability 1-p and uniform on [0,1] with probability p.

  • (a) Dedicated: W=[1,0], b=(0,b_2) with the best b_2\ge0. Show that the loss is r\,(p/3-p^2/4).
  • (b) Antipodal: W=[1,-1], b=0. Show that the loss is (1+r)\,p^2/6.
  • (c) Show that antipodal storage wins iff p<4r/(2+5r). What does this say about sparse features, and about dense ones?
Solution

(a) \hat x_1=\mathrm{ReLU}(x_1)=x_1, so feature 1 costs nothing. \hat x_2=\mathrm{ReLU}(b_2)=b_2 is a constant; the best constant is \mathbb Ex_2=p/2, leaving r\operatorname{Var}x_2=r(\mathbb Ex_2^2-(\mathbb Ex_2)^2)=r(p/3-p^2/4).

(b) W^\top W=\begin{pmatrix}1&-1\\-1&1\end{pmatrix}, so \hat x_1=\mathrm{ReLU}(x_1-x_2) and \hat x_2=\mathrm{ReLU}(x_2-x_1). If at most one feature is active, reconstruction is exact: the ReLU erases the negative interference. If both are active and x_1>x_2, then \hat x_1=x_1-x_2 and \hat x_2=0, an error of (1+r)x_2^2; symmetrically otherwise. So the loss is p^2(1+r)\,\mathbb E[\min(x_1,x_2)^2]. The minimum of two uniforms has density 2(1-s) on [0,1], so \mathbb E\min^2=\int_0^12s^2(1-s)\,ds=1/6.

(c) (1+r)p^2/6<r(p/3-p^2/4). Divide by p/12: 2(1+r)p<4r-3rp, i.e. p(2+5r)<4r. Sparse features (p\to0) are superposed for any r>0, even when one matters little, because interference costs O(p^2) against O(p) for dropping a feature. Dense features force a choice. At r=1 the threshold is p=4/7. This is the dashed boundary in panel 1 of the figure.

9. Uneven dependence on weights and initialization: critical modules and lottery tickets

After training, most layers can be reset to their initial weights with almost no loss, but a few cannot; and trained networks contain sparse subnetworks that, rewound to their initial weights, train alone to full accuracy but do worse with fresh random weights: the trained function depends very unevenly on its weights, and on the particular initialization.

Lottery ticket hypothesis, Wikipedia Importance ●●●●● Length ●●○○○

The hypothesis, iterative magnitude pruning with rewinding, and the follow-up literature in one place; the papers below are the originals.

Supplemental

Are all layers created equal?, Zhang, Bengio & Singer (2019; JMLR 2022) Importance ●●●●○ Length ●●●○○

Resets one layer at a time to its initial or to fresh random weights: most layers are “robust” to re-initialization, a few are “critical”, and which ones depends on the architecture.

The lottery ticket hypothesis: finding sparse, trainable neural networks, Frankle & Carbin (ICLR 2019) Importance ●●●●○ Length ●●●○○

Iterative magnitude pruning with rewinding to initialization finds subnetworks under 10–20% of the original size that match full accuracy on MNIST and CIFAR-10; re-randomized, they do worse.

Linear mode connectivity and the lottery ticket hypothesis, Frankle, Dziugaite, Roy & Carbin (ICML 2020) Importance ●●●○○ Length ●●●○○

Tickets work only from the point where training becomes stable to SGD noise (two runs with different data orders end up linearly connected): at initialization for small networks, early in training for ResNet-50.

10. Distillation

A small network trained on a large network’s softened outputs learns far more than from the labels, even about a class it never sees, so the probabilities assigned to wrong answers carry much of what the teacher knows.

Knowledge distillation, Wikipedia Importance ●●●●● Length ●●○○○

What distillation is, the temperature-softened objective, and where it is used; Hinton et al. below is the original paper.

Supplemental

Distilling the knowledge in a neural network, Hinton, Vinyals & Dean (2015) Importance ●●●●○ Length ●●○○○

Temperature-softened teacher outputs transfer most of a large model’s or ensemble’s performance; a student whose transfer set omits every 3 still classifies most test 3s correctly.

11. Hessian outliers: gradient descent in a tiny subspace and the spectrum

After a short initial phase the gradient lies almost entirely in the span of a few top Hessian eigenvectors, about as many as there are classes, in networks with millions of parameters; the spectrum itself is a bulk near zero plus those few outliers, which appear early in training and which batch normalization largely removes.

Gradient descent happens in a tiny subspace, Gur-Ari, Roberts & Dyer (2018) Importance ●●●●● Length ●●○○○

Measures the gradient’s overlap with the top Hessian eigenspace during training; the subspace is small, changes slowly, and a short toy model explains the count.

Supplemental

An investigation into neural net optimization via Hessian eigenvalue density, Ghorbani, Krishnan & Xiao (ICML 2019) Importance ●●●●○ Length ●●●○○

Estimates full Hessian spectra up to ImageNet scale with stochastic Lanczos; without batch norm, early outliers capture the gradient, and their absence with batch norm is linked to faster optimization.

Exercises: where the gradient is versus where learning happens; scale invariance under batch norm

Exercise 11.1 (a quadratic toy). This toy is not the paper’s model. It shows that “the gradient lies in the top subspace” and “the loss goes down along the top subspace” can come apart. Take L(\theta)=\tfrac12(\theta-\theta^*)^\top H(\theta-\theta^*) with H symmetric positive semi-definite, orthonormal eigenvectors v_i with eigenvalues \lambda_1\ge\lambda_2\ge\dots, and write \theta-\theta^*=\sum_i c_iv_i. The overlap is \lVert P_kg\rVert^2/\lVert g\rVert^2, where g=\nabla L and P_k projects onto v_1,\dots,v_k.

  • (a) Show that the overlap equals \sum_{i\le k}\lambda_i^2c_i^2\big/\sum_i\lambda_i^2c_i^2, and that L=\tfrac12\sum_i\lambda_ic_i^2.
  • (b) For gradient descent \theta\leftarrow\theta-\eta g, show that c_i(t)=(1-\eta\lambda_i)^t\,c_i(0).
  • (c) Take one outlier \lambda_1=\Lambda and p-1 bulk directions with \lambda_i=\mu, where 0<\mu\le\Lambda/2, and k=1. Assume c_1(0)\neq0 and some bulk c_i(0)\neq0. For which step sizes 0<\eta<2/\Lambda does the overlap tend to 1 as t\to\infty?
  • (d) Take \Lambda=100, \mu=1, p=1000, \eta=0.0199 and all c_i(0)=1. At t=0 and at t=100, compute the overlap and the fraction of the one-step loss decrease that comes from the bulk.
Hint

For (d): direction i contributes \tfrac12\lambda_ic_i^2\bigl(1-(1-\eta\lambda_i)^2\bigr) to the decrease in one step.

Solution

(a) g=H(\theta-\theta^*)=\sum_i\lambda_ic_iv_i and P_kg=\sum_{i\le k}\lambda_ic_iv_i. By orthonormality, \lVert g\rVert^2=\sum_i\lambda_i^2c_i^2 and \lVert P_kg\rVert^2=\sum_{i\le k}\lambda_i^2c_i^2. Likewise L=\tfrac12\sum_i\lambda_ic_i^2.

(b) \theta-\theta^*\leftarrow(I-\eta H)(\theta-\theta^*), and v_i is an eigenvector of I-\eta H with eigenvalue 1-\eta\lambda_i. So c_i\leftarrow(1-\eta\lambda_i)c_i at each step.

(c) Let r_1=|1-\eta\Lambda| and r_b=1-\eta\mu. Since \eta<2/\Lambda and \mu\le\Lambda/2, 0<r_b<1. By (a) and (b), with k=1, \text{overlap}(t)=\frac{\Lambda^2c_1(0)^2\,r_1^{2t}}{\Lambda^2c_1(0)^2\,r_1^{2t}+\mu^2\sum_{i\ge2}c_i(0)^2\,r_b^{2t}}, which tends to 1 if r_1>r_b and to 0 if r_1<r_b. If \eta\Lambda\le1, then r_1=1-\eta\Lambda<r_b, so the overlap goes to 0. If \eta\Lambda>1, then r_1=\eta\Lambda-1, and r_1>r_b iff \eta(\Lambda+\mu)>2. So the overlap tends to 1 exactly for \frac{2}{\Lambda+\mu}<\eta<\frac{2}{\Lambda}, a narrow window just below the stability limit, where the outlier component flips sign every step and shrinks slowly. At \eta=2/(\Lambda+\mu) exactly, r_1=r_b and the overlap stays at its initial value.

(d) r_1=0.99 and r_b=0.9801; \eta=0.0199>2/101\approx0.0198, so we are in the window.

  • t=0: overlap =10^4/(10^4+999)\approx0.909. One-step decrease: outlier \tfrac12\cdot100\cdot(1-0.9801)=0.995; bulk \tfrac12\cdot999\cdot(1-0.9801^2)\approx19.68. The bulk supplies about 95\%.
  • t=100: c_1^2=0.99^{200}\approx0.134 and bulk c_i^2=0.9801^{200}\approx0.0180. Overlap =1340/(1340+17.9)\approx0.987. Decrease: outlier \approx0.133, bulk \approx0.353. The bulk still supplies about 73\%.

So an overlap near 1 is compatible with most of the loss decrease coming from outside the top subspace: the gradient is dominated by a large oscillating component that contributes little progress. (Here the outlier eventually dominates the decrease too, after about 150 steps, since it decays slowest.) The overlap measures where the gradient is, not where learning happens; whether real training behaves like this takes intervention experiments on real networks.

Exercise 11.2 (scale invariance). This calculation is not in the paper; it shows one concrete way batch normalization changes curvature. With batch statistics (and \varepsilon=0, nonzero pre-activation variance), rescaling a unit’s nonzero incoming weight vector w by \alpha>0 leaves its normalized output unchanged. So, with all other parameters fixed, L(\alpha w)=L(w) for every \alpha>0; assume L is twice differentiable near w, and write g=\nabla L(w), H=\nabla^2L(w) with respect to w.

  • (a) Show that w^\top g=0.
  • (b) Show that \nabla L(\alpha w)=\alpha^{-1}\nabla L(w) and \nabla^2L(\alpha w)=\alpha^{-2}\nabla^2L(w).
  • (c) Show that Hw=-g. Deduce that w^\top Hw=0. When is w an eigenvector of H with eigenvalue 0, and why is it not one in general?
  • (d) For a gradient step w'=w-\eta g, show that \lVert w'\rVert^2=\lVert w\rVert^2+\eta^2\lVert g\rVert^2. Using (b), separate the effect of changing \lVert w\rVert from that of changing the direction u=w/\lVert w\rVert. What effective step size acts on the direction?

As with layer rescaling in ReLU networks (Dinh et al., Sharp minima can generalize, in MDL1, section ‘Regularization: prior, constraint, filter, flatness’), curvature measured in these weights is not a property of the function alone.

Solution

(a) Differentiate L(\alpha w)=L(w) with respect to \alpha: w^\top\nabla L(\alpha w)=0 for all \alpha>0. At \alpha=1, w^\top g=0.

(b) Differentiate L(\alpha w)=L(w) with respect to w: \alpha\nabla L(\alpha w)=\nabla L(w). Differentiate again: \alpha^2\nabla^2L(\alpha w)=\nabla^2L(w).

(c) Differentiate \nabla L(\alpha w)=\alpha^{-1}\nabla L(w) with respect to \alpha: \nabla^2L(\alpha w)\,w=-\alpha^{-2}\nabla L(w). At \alpha=1, Hw=-g, so w^\top Hw=-w^\top g=0.

If g=0 (a stationary point), Hw=0 and w is an eigenvector with eigenvalue 0. If g\neq0, Hw=-g is nonzero and orthogonal to w, so w is not an eigenvector: H couples the radial direction to the gradient direction. Away from a minimum H need not be positive semi-definite, so w^\top Hw=0 does not force Hw=0.

(d) \lVert w-\eta g\rVert^2=\lVert w\rVert^2-2\eta\,w^\top g+\eta^2\lVert g\rVert^2=\lVert w\rVert^2+\eta^2\lVert g\rVert^2 by (a): without weight decay the norm never decreases. By (b) with \alpha=\lVert w\rVert, \nabla L(w)=\lVert w\rVert^{-1}\nabla L(u) and H(w)=\lVert w\rVert^{-2}H(u). To first order u'\approx u-(\eta/\lVert w\rVert^2)\nabla L(u), so the effective step size is \eta/\lVert w\rVert^2. Norm growth shrinks this factor and the radial factor in H(w_t)=\lVert w_t\rVert^{-2}H(u_t); it does not guarantee the Hessian eigenvalues fall, because H(u_t) changes too. Weight decay works against the norm growth.

12. Low-rank RL updates and the KL leash

RL fine-tuning of LLMs is reported to change few parameters yet at full rank, or to be dominated by one rank-1 direction, and to forget less than SFT; how these fit together, and how much is a bfloat16 artifact, is open (KL-regularized optimum: MDL.X (deprecated), section ‘From Bellman to AlphaZero’, exercises ‘planning as inference’).

Reinforcement learning finetunes small subnetworks in large language models, Mukherjee, Yuan, Hakkani-Tür & Peng (NeurIPS 2025) Importance ●●●●● Length ●●●○○

Across 7 RL algorithms and 10 LLMs, RL changes 5–30% of parameters, nearly full-rank per matrix, overlapping across seeds and algorithms; fine-tuning that subnetwork alone gives a nearly identical model.

RL’s Razor: why online reinforcement learning forgets less, Shenfeld, Pari & Agrawal (2025) Importance ●●●●● Length ●●●○○

On-policy RL forgets less than SFT at matched new-task performance, with forgetting predicted by KL to the base policy; their Section 6 finds the update sparsity was a bfloat16 effect.

Supplemental

Also: On predictability of reinforcement learning dynamics for large language models, Cai et al. (2025): the top singular direction of each RL update matrix recovers over 99% of the reasoning gains and evolves almost linearly, so the final update can be extrapolated from an early window.

Exercises: sparse updates and bfloat16

Exercise 12.1 (sparse is not low-rank). An update \Delta W\in\mathbb R^{n\times n} is sparse if most entries are zero, and has rank at most r if \Delta W=AB^\top with A,B\in\mathbb R^{n\times r} (LoRA’s form, with small r).

  • (a) Give a matrix with only n nonzero entries and rank n, and a matrix of rank 1 with all n^2 entries nonzero.
  • (b) Fix a support S\subseteq\{1,\dots,n\}^2 that contains the positions \{(i,\pi(i))\} of some permutation \pi. Show that for almost every choice of real values on S (zero elsewhere), the matrix is invertible.
  • (c) What does (b) suggest about how surprising “sparse but nearly full-rank” is? You may use that a random support including each position independently with probability \rho contains some permutation pattern with high probability once \rho n\gg\log n.
Hint

For (b): \det is a polynomial in the entries on S; find one choice of values where it is nonzero.

Solution

(a) The identity I_n (or any scaled permutation matrix) has n nonzero entries and rank n. For u,v\in\mathbb R^n with no zero entries, uv^\top has all entries u_iv_j\neq0 and rank 1.

(b) \det M=\sum_\sigma\operatorname{sgn}(\sigma)\prod_iM_{i\sigma(i)} is a polynomial in the entries on S. Putting 1 on the positions (i,\pi(i)) and 0 elsewhere on S gives the permutation matrix of \pi, with determinant \pm1, so the polynomial is not identically zero. The zero set of a nonzero polynomial has Lebesgue measure zero, so \det M\neq0 for almost every choice of values.

(c) Sparsity alone does not imply low rank: generic values on a support containing a permutation pattern give full rank almost surely, and a support with density \rho=0.05 at width n=4096 has \rho n\approx200\gg\log n\approx8. So “sparse but nearly full-rank” is what a generic sparse update looks like. Learned supports and values can be structured, so the measured rank is still an empirical fact, but the informative findings are the small fraction of parameters that change (if it survives a higher-precision measurement, Exercise 12.2) and its overlap across runs. A rank-r update is a different restriction: it is generically dense.

Exercise 12.2 (what “unchanged” means). Mukherjee et al. count a parameter as unchanged when its bfloat16 values before and after fine-tuning differ by at most 10^{-5}. A bfloat16 number has 8 exponent bits and 7 stored fraction bits (8 significant bits with the implicit leading 1).

  • (a) Show that for normal values with 2^e\le|w|<2^{e+1}, the spacing between neighbouring bfloat16 numbers is 2^{e-7}.
  • (b) For which magnitudes |w| can a change of the stored value be within the 10^{-5} tolerance?
  • (c) Suppose training keeps a higher-precision copy that starts exactly at the bfloat16 value w (|w|>2^{-9}, not a power of two), and the final weight w+\Delta is rounded to the nearest bfloat16 number. Roughly how large must |\Delta|/|w| be for the parameter to count as updated? Evaluate for |w|\approx0.02.
  • (d) Measured the same way, SFT leaves only 6–15% of parameters unchanged, against roughly 70–95% for RL. What does this comparison rule out, and what does it not?
Solution

(a) The bfloat16 numbers in [2^e,2^{e+1}) are 2^e(1+f/2^7) for f=0,\dots,127, so neighbours differ by 2^{e-7}.

(b) Away from powers of two, a change of stored value is at least one spacing, 2^{e-7}, which is \le10^{-5} only if e\le-10, i.e. |w|<2^{-9}\approx0.002. At |w|=2^{-9} exactly, the neighbour toward zero is 2^{-17}\approx7.6\times10^{-6} away (within tolerance) and the one away from zero 2^{-16}\approx1.5\times10^{-5} (outside). So a distinct stored value lies within tolerance iff |w|\le2^{-9}. Above that, “unchanged” means the stored bfloat16 value is identical.

(c) Rounding returns w when |\Delta| is below half a spacing, 2^{e-8}, which lies between |w|/512 and |w|/256: roughly 0.2–0.4% of |w|. For |w|\approx0.02, e=-6, so the threshold is 2^{-14}\approx6.1\times10^{-5}, about 0.3%. “Unchanged” does not mean “received no gradient”. If updates were written straight into bfloat16 weights every step, the same threshold would apply per step, and many small steps could all be lost.

(d) It rules out the simplest artifact, that this measurement makes every fine-tune look sparse: SFT looks dense under it. It does not distinguish “RL moves few parameters” from “RL moves most parameters by less than the rounding threshold”, since SFT may simply move weights further. That needs a higher-precision measurement. Shenfeld et al. (RL’s Razor, Section 6) did this in their setting: float32 training gave the same performance and no sparsity. Their setting is not Mukherjee et al.’s, so this does not settle it everywhere, but the concern is real. Mukherjee et al.’s finding that tuning only the subnetwork gives a nearly identical model bears on whether sub-threshold movement matters for the function.

13. Physics of language models

Transformers trained on a deep, ambiguous synthetic grammar learn to parse it, though nothing in the architecture is built for parsing (data-model side: MDL3, section ‘Hierarchical representations’).

Physics of language models: Part 1, learning hierarchical language structures, Allen-Zhu & Li (2023; project page, with the later parts) Importance ●●●●● Length ●●●○○

GPT-2-sized models trained on context-free grammars generate grammatical strings; linear probes recover the parse, and attention patterns match what a dynamic-programming parser needs.

14. Hidden signals: low-rank logits and subliminal learning

Language models’ multi-token log-probabilities are approximately low-rank and share structure across models, though the architecture forces low rank only for the next token; and a student fine-tuned on a teacher’s innocuous outputs picks up the teacher’s traits.

Subliminal learning: language models transmit behavioral traits via hidden signals in data, Cloud, Le, Chua, Betley, Sztyber-Betley, Hilton, Marks & Evans (Anthropic Alignment Science blog, 2025; paper) Importance ●●●●● Length ●●○○○

Traits (a favourite animal, misalignment) pass through filtered numbers, code or chain-of-thought, but only when teacher and student share an initialization; a short argument shows why, to first order.

Supplemental

Sequences of logits reveal the low rank structure of language models, Golowich, Liu & Shetty (2025) Importance ●●●●○ Length ●●●○○

Prompt-by-continuation log-probability matrices are approximately low-rank, with linear relations among prompts consistent across models; this lets one answer a target prompt from outputs on unrelated or nonsense prompts.

Subliminal effects in your data: a general mechanism via log-linearity, Aden-Ali, Golowich, Liu, Shetty, Moitra & Haghtalab (2026) Importance ●●●○○ Length ●●●○○

If log-probabilities are roughly bilinear in shared embeddings, selecting preference data by projection onto a trait direction instils the trait through DPO, within a model and more weakly across architectures.

Also: Subliminal learning is a LoRA artifact (Nief et al. 2026; its arXiv listing says follow-up work showed some main claims incorrect, and Cloud et al.’s MNIST experiment uses no LoRA).