On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime

Shuai Jiang, Alexey Voronin, Eric Cyr +1

Spectral bias, the tendency of neural networks to learn low frequencies first, can be both a blessing and a curse. While it enhances the generalization capabilities by suppressing…