Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

Abstract

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• Striving for Simplicity: The All Convolutional Net
• SGDR: Stochastic Gradient Descent with Warm Restarts
• The Loss Surfaces of Multilayer Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• New insights and perspectives on the natural gradient method
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Three Factors Influencing Minima in SGD
• L2 Regularization versus Batch and Weight Normalization
• Sharp Minima Can Generalize For Deep Nets
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Norm matters: efficient and accurate normalization schemes in deep networks
• Three Mechanisms of Weight Decay Regularization
• Adaptive Regularization of Some Inverse Problems in Image Analysis

Cited by in corpus (3)

• Review: Deep Learning in Electron Microscopy
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Better Schedules for Low Precision Training of Deep Neural Networks