Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise

Jiayu Zhang, Tianyi Lin

A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized. The layerwise input-output structure of neural networks mot…