Volumization as a Natural Generalization of Weight Decay

Abstract

We propose a novel regularization method, called \textit{volumization}, for neural networks. Inspired by physics, we define a physical volume for the weight parameters in neural networks, and we show that this method is an effective way of regularizing neural networks. Intuitively, this method interpolates between an $L_2$ and $L_{\infty }$ regularization. Therefore, weight decay and weight clipping become special cases of the proposed algorithm. We prove, on a toy example, that the essence of this method is a regularization technique to control bias-variance tradeoff. The method is shown to do well in the categories where the standard weight decay method is shown to work well, including improving the generalization of networks and preventing memorization. Moreover, we show that the volumization might lead to a simple method for training a neural network whose weight is binary or ternary.

arXiv comment18 pages, 20 figures

References in corpus (13)

• Decoupled Weight Decay Regularization
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Survey on Methods and Theories of Quantized Neural Networks
• Regularization for Deep Learning: A Taxonomy
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Three Mechanisms of Weight Decay Regularization
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Learning Not to Learn in the Presence of Noisy Labels
• LaProp: Separating Momentum and Adaptivity in Adam