Spectral filters, and where a scaling exponent comes from

Regularization and early stopping keep strong directions and kill weak ones (A). Count how many directions survive at sample size n, and the slope of the learning curve is set by the spectrum (B).

A. Ridge and stopped gradient flow as filters on the eigenvalues

WB2 §4. Loss (1/2n)‖Xβ−y‖², eigen-decomposition XᵀX = V diag(sk²) Vᵀ, fit β̂ = Σk φk β̂OLSk vk. Both curves are exact formulas.

ridge φ = s²/(s²+nλ) gradient flow from 0, stopped at t: φ = 1 − e−s²t/n power-law eigenvalues sk²/n = k−α (α from panel B)

B. Test loss vs n for ridge on a power-law spectrum

WB5 §1. Gaussian features with covariance eigenvalues λk = k−α; target y = Σ βkxk + ε, with per-mode signal power ck² = λkβk² ∝ k−α′ (normalized Σck² = 1), so the unresolved tail Σk>K ck² ≈ K−(α′−1)/((α′−1)ζ(α′)); noise variance σ². Infinite number of features.

deterministic equivalent, ridge at optimal λ heuristic: resolve the top K ≤ n modes ○ Monte Carlo ridge (p = 5·10⁴ features, 16 draws; ±1 s.e.) slope −(predicted exponent)
What to look at. In A, press “match half-points”: the two filters now cross ½ at the same eigenvalue, yet gradient flow is sharper. In the weak tail both are linear in s² (ridge ≈ s²/nλ, flow ≈ s²t/n, lower by the factor ln 2), so ridge passes about 1/ln2 ≈ 1.44 times more of each weak direction. In the strong directions ridge still shrinks by about nλ/s², while flow’s shrinkage e−s²t/n vanishes exponentially. The difference strip shows exactly where they disagree (at most about 0.14 when matched). The naive matching t = 1/λ leaves them offset by a factor ln 2. The effective degrees of freedom Σφk counts how many of the power-law eigenvalues (ticks) survive. In B, that count is the whole story to first order: with n samples about the top n modes are resolved (fewer once noise forces regularization), so the excess loss is the unresolved tail, ∝ n−(α′−1) without noise and ∝ n−(α′−1)/α′ at large n with noise and optimal λ. The deterministic-equivalent approximation (checked against Monte Carlo) has the same slope as the heuristic but a different prefactor. When α′−1 > 2α, ridge saturates at exponent 2α and the heuristic is wrong. At the boundary α′−1 = 2α, logarithmic corrections accompany the power law. What it does not show. This is a linear model (ridge/kernel regression) with a spectrum we chose. The exponent comes from the data spectrum and the target’s alignment with it (α′, not α, unless the target is generic). It explains the form of a learning curve, not what a network learns or why real data has this spectrum; a different mechanism, such as discrete quanta, can produce the same curve (cf. “Power laws are not enough”). Panel A’s ticks are idealized: they treat the empirical eigenvalues sk²/n as equal to the population ones.
Formulas and checks

A. Ridge minimizes (1/2n)‖Xβ−y‖² + (λ/2)‖β‖², so β̂ = (XᵀX+nλI)⁻¹Xᵀy. Gradient flow dβ/dt = −(1/n)Xᵀ(Xβ−y) from β(0)=0 gives φk(t) = 1−e−sk²t/n. Half-points: ridge at s²/n = λ, flow at s²/n = ln2/t. df = Σkφk is summed over k = 1…∞ with sk²/n = k−α.

B. Ridge with the same nλ convention, population features. Effective regularization κ solves κ(1 − df₁/n) = λ, with df₁ = Σλk/(λk+κ) and df₂ = Σλk²/(λk+κ)². Excess test loss E = (κ²Σck²/(λk+κ)² + σ²)/(1 − df₂/n) − σ² (Hastie–Montanari–Rosset–Tibshirani; Bach 2023; Simon et al. “eigenlearning”). λ is optimized numerically over λ ≥ 0. Heuristic: Eheur = minK≤n[Σk>Kck² + σ²K/n]. Predicted exponent: s = min(α′−1, 2α) without noise, and s/(s+1) with noise.

Checks (checks/spectral_filters_check.py): the filter formulas match direct linear algebra and small-step GD to within 1e-6. The DE matches Monte Carlo ridge to within 1–7% for n = 16…800 at several settings. The DE’s slopes over n ∈ [10⁴,10⁵] match the predicted exponents (1.000, 2.000, 0.500, and 3.000 in the saturated case α=1.5, α′=5, where the heuristic says 4). This page recomputes reference values on load and logs the comparison to the console.