Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks

Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang +3

L2 regularization for weights in neural networks is widely used as a standard training trick. However, L2 regularization for gamma, a trainable parameter of batch normalization, re…