Logit Attenuating Weight Normalization

Abstract

Over-parameterized deep networks trained using gradient-based optimizers are a popular choice for solving classification and ranking problems. Without appropriately tuned ${\ell }_2$ regularization or weight decay, such networks have the tendency to make output scores (logits) and network weights large, causing training loss to become too small and the network to lose its adaptivity (ability to move around) in the parameter space. Although regularization is typically understood from an overfitting perspective, we highlight its role in making the network more adaptive and enabling it to escape more easily from weights that generalize poorly. To provide such a capability, we propose a method called Logit Attenuating Weight Normalization (LAWN), that can be stacked onto any gradient-based optimizer. LAWN controls the logits by constraining the weight norms of layers in the final homogeneous sub-network. Empirically, we show that the resulting LAWN variant of the optimizer makes a deep network more adaptive to finding minimas with superior generalization performance on large-scale image classification and recommender systems. While LAWN is particularly impressive in improving Adam, it greatly improves all optimizers when used with large batch sizes

arXiv comment23 pages

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• ADADELTA: An Adaptive Learning Rate Method
• Deep Residual Learning for Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Large Batch Training of Convolutional Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Measuring the Effects of Data Parallelism on Neural Network Training
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
• Minnorm training: an algorithm for training over-parameterized deep neural networks
• Theory III: Dynamics and Generalization in Deep Networks