Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov +1

A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of s…