Hyperball May Not Be a Free Lunch

Yihao Xiao, Jialong Sun, Zitian Gao +5

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing…