Superposition in a toy autoencoder

The toy model of Elhage et al., "Toy Models of Superposition" (2022), in their notation: features x ∈ ℝⁿ, each x_i = 0 with probability S (the sparsity) and otherwise ~ Uniform[0,1]; reconstruction x′ = ReLU(W^⊤W x + b) with W ∈ ℝ^m×n; loss L = 𝔼 Σ_i I_i(x_i − x′_i)². Column W_i ∈ ℝ^m is where feature i is stored.

1 · Two features, one dimension: when does superposition win?

n = 2, m = 1, importances I = (1, r). Drag the dot. Background: which candidate has the lowest exact expected loss.

feature 2 ignored feature 1 ignored antipodal superposition exact boundary closed form, b=0, w=±1

show trained models (Adam, best of 6 inits; ring = phase reached by training, ✕ = disagrees with theory)

Point

Winning solution: columns of W on the one hidden axis

Expected loss of each candidate exact

What to look at. Candidates: dedicate the dimension to one feature (W = [1, 0], bias b₂ = 𝔼x₂; loss r(p/3 − p²/4) with density p = 1−S), or store both antipodally (W = [a, −c]). With a = c = 1, b = 0 the antipodal loss is exactly (1+r)p²/6: interference costs only when both features fire (probability p²), and the ReLU erases the negative interference when only one fires. Comparing gives the dashed boundary p* = (r/3)/((1+r)/6 + r/4) for r ≤ 1. The solid boundary optimises a, c, b₁, b₂ (closed-form expectations, numerical minimisation); for r far from 1 the optimum has |w₂| < |w₁|, and Adam-trained models (dots) agree with it in 151 of 153 cells. Sparse features get superposed even when they matter little; dense ones force a choice. Compare the paper's theoretical and empirical n=2, m=1 phase diagrams in "Superposition as a Phase Change".
Not shown. Only these candidate families are compared (training finds no others here); near p = 1, r = 1 all three are within 0.1% and the label is not meaningful. This is a toy autoencoder with a ReLU readout, not a trained network.

2 · Five features in two dimensions, trained in your browser

m = 2, importances I_i = dⁱ⁻¹. Arrows are the columns W_i, darker = more important. simulation

Sparsity S

Importance decay d

Features n Seed

What to look at. Each run is Adam (lr 0.01, cosine decay) for 3000 steps on minibatches of 1024 freshly sampled sparse vectors, from a seeded random init, so a given (S, d, n, seed) always gives the same picture. At S = 0 only the m = 2 most important features get (orthogonal) dimensions and the rest are dropped, like a PCA. As S grows the model first stores pairs antipodally (a square, 4 features), then all five as a regular pentagon at 72°: more features than dimensions, read out cleanly by the ReLU with a negative bias because two features rarely fire together. Try other seeds near the transitions (e.g. S ≈ 0.9): training lands in different local optima.
Not shown. This is a toy autoencoder with a ReLU readout and independent, uniformly distributed features. Whether real networks operate in this regime (many sparse features, few dimensions) is the empirical question; if they do, small input perturbations along the interference directions are a natural route to adversarial examples (WB6 §5).