When Does ${\ell }_2$-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ${\ell }_1$ Implicit Bias

Abstract

Benign overfitting is well-characterized in ${\ell }_2$ geometries, but its behavior under the ${\ell }_1$ implicit bias of greedy ensembles remains challenging. The analytical barrier stems from the non-linear coupling of coordinate selection thresholds, which invalidates standard spectral resolvent tools. To isolate this algorithmic bias, we characterize the high-dimensional risk of continuous-time ${\ell }_2$-Boosting over $p$ features and $n$ samples. By coupling the Convex Gaussian Minimax Theorem with delicate asymptotic expansions of double-sided truncated Gaussian moments, we analytically resolve the non-smooth ${\ell }_1$ interpolant. Under an isotropic pure-noise model, we prove that benign overfitting fails at the linear rate: greedy selection localizes noise into sparse active sets, and the excess variance decays at a logarithmic rate $\hat{I}˜(\ddot{I}{ƒ}^2/\log (p/n))$ for noise variance $\ddot{I}{ƒ}^2$. We remark that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem. For spiked-isotropic designs with $k^{\ast }$ head eigenvalues and $r_2=p-k^{\ast }$ tail dimensions, the risk converges to zero when $r_2\gg n$, but only at a logarithmic rate $\hat{I}˜(\ddot{I}{ƒ}^2/\log (r_2/n))$, which is slower than the linear decay observed in ${\ell }_2$ geometries. To avoid this slow convergence, we analyze the non-smooth subdifferential dynamics of the boosting flow. This yields a tuning-free early stopping rule that, under a bounded ${\ell }_1$-path condition, recovers the Lasso basic inequality and attains the minimax-optimal empirical prediction rate for ${\ell }_1$-bounded signals.