Mode explorer: Saxe dynamics with imbalance and depth

Each singular mode of the target Σyx = Σα sα uαvα⊤ evolves on its own under gradient flow from decoupled initial conditions. Small, balanced initialization gives sigmoids ordered by s, with plateaus. Large imbalance between layers (d ≫ s and a₀) removes the plateaus and makes every mode relax exponentially at a nearly common rate (close to d for two layers, larger for deeper ones); intermediate d gives a mix. For small a₀, extra depth lengthens the plateaus.

Presets:

A. Mode strengths aα(t)

Time in units of τ = 1/η, so t/τ = η × (number of steps). Dotted lines: targets sα. Dots: half-times (a = s/2).

B. Effective kernel kα(t), the rate in τȧ = k(s − a)

kα = Σj Πi≠j wi² is the end-to-end NTK restricted to mode α. Flat means lazy (frozen kernel); growing by orders of magnitude means rich.

The model. (Depth and the kernel panel are explained in the notebook's §2 and §4.) For depth N, mode α is carried by one scalar per layer, w1,…,wN, with a = w1⋯wN and τẇj = (Πi≠j wi)(s − a). The differences wj+1² − wj² are conserved; the slider sets them all equal to d (for N = 2 this is the notebook's d = c² − b²), so at N ≥ 3 a large d makes every layer but the first large. Only |d| matters for a(t), so d ≥ 0 here. N = 1 is the shallow model τȧ = s − a.

Two layers. The curves obey τȧ = √(4a² + d²)(s − a). The same equation holds, with d replaced by the mode invariant Δ (imbalance including misalignment, notebook §4.3), for a mode whose hidden-layer weights do not overlap other modes'; for wide random initializations it holds approximately, with Δ ≈ 2σ²h for Gaussian weights of variance σ² at width h.

Closed forms. With d = 0, a(t) = s e2st/τ/(e2st/τ − 1 + s/a0), with half-time (τ/2s) ln(s/a0 − 1). For balanced N ≥ 3, τȧ = N a2−2/N(s − a), and the plateau grows like a0−(1−2/N) instead of ln(1/a0).

How the curves are computed Adaptive fourth-order Runge–Kutta on the per-layer scalars, step ≈ 0.02 divided by the current rate. The table compares the numerical half-time with the N = 2 closed form when it applies. The check script checks/mode_explorer_check.py runs this page's math block in node and compares it with scipy and with the closed forms.