Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer

Li Xiang, Chen Shuo, Xia Yan +1

The merits of fast convergence and potentially better performance of the weight normalization family have drawn increasing attention in recent years. These methods use standardizat…