Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

Ziquan Liu, Yufei Cui, Antoni B. Chan

Using weight decay to penalize the L2 norms of weights in neural networks has been a standard training practice to regularize the complexity of networks. In this paper, we show tha…