Proving the t^(−2β/(1+β)) feature-learning loss exponent for hard tasks under power-law kernel spectra

Proved in the lazy and linear regimes. Kernel ridge regression with power-law spectra has minimax-optimal rates (Caponnetto and De Vito, 2007). For random-feature models trained by one-pass SGD with power-law covariance, Paquette, Paquette, Xiao and Pennington (arXiv:2405.15074, 2024) prove loss asymptotics and identify the phases of compute-optimal scaling; Lin, Wu, Kakade, Bartlett and Lee (arXiv:2406.08466, 2024) prove data, parameter and compute scaling for SGD on linear regression with power-law spectra.

Proved with feature learning, but on isotropic data. Ren, Nichani, Wu and Lee (arXiv:2504.19983, NeurIPS 2025) analyse online SGD for a two-layer student learning $f_*(x)=\sum_{p\le P}a_p\sigma(\langle x,v_p^*\rangle)$ with , orthonormal , an even activation of information exponent above 2 and ; they prove sharp recovery times for each direction and derive scaling exponents in samples, SGD steps and width. Ben Arous, Erdogdu, Vural and Wu (arXiv:2508.03688, NeurIPS 2025) prove scaling laws for quadratic-activation networks with extensive width and power-law second-layer coefficients. In both, the power law comes from the target's coefficient decay over orthogonal directions of isotropic data, so the hard-task mechanism above (feature learning against a power-law input spectrum) is not covered.

Direct benefit is limited: a proof would tell practitioners which exponents depend on architecture-level feature learning and which on data spectra, informing compute-allocation rules, but practical scaling-law fits will remain empirical. The methods (high-dimensional SGD dynamics with evolving kernels) are reusable in the theory of training dynamics.

Resolved by a proof of (ii): matching upper and lower bounds on the loss exponent of a nonlinear two-layer network trained by online SGD in the mean-field parametrisation on power-law-spectrum data, with the hard-task exponent identified explicitly (whether or not it equals $2\beta/(1+\beta)$). A rigorous proof of (i) for the Bordelon–Atanasov–Pehlevan model alone, or a rigorous counterexample to its predicted exponent, is major partial progress.