Data augmentation instead of explicit regularization

Abstract

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.

arXiv commentMajor changes: 1. updated figures; 2. statistical significance analysis of results through bootstrap; 3. information about carbon emissions produced with the experiments; 4. extended discussion about the need for weight decay and dropout; 5. literature review updated; 6. slight update of the definitions of explicit and implicit regularization, with several examples for illustration

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• An Analysis of Deep Neural Network Models for Practical Applications
• Data Augmentation Generative Adversarial Networks
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• Fractional Max-Pooling
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Dropout Training as Adaptive Regularization
• Regularization for Deep Learning: A Taxonomy
• Quantifying the Carbon Emissions of Machine Learning
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• Dropout as data augmentation
• A Group-Theoretic Framework for Data Augmentation
• Implicit Regularization in Deep Learning
• Dataset Augmentation in Feature Space
• Big Neural Networks Waste Capacity
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Do deep nets really need weight decay and dropout?
• Adversarial Training Versus Weight Decay
• Generalisation dynamics of online learning in over-parameterised neural networks
• Does Data Augmentation Lead to Positive Margin?

Cited by in corpus (25)

• Unsupervised Data Augmentation for Consistency Training
• Automated brain extraction of multi-sequence MRI using artificial neural networks
• A Survey on Data Augmentation for Text Classification
• Dataset Condensation with Differentiable Siamese Augmentation
• Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Fair Mixup: Fairness via Interpolation
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Cross-directional Feature Fusion Network for Building Damage Assessment from Satellite Imagery
• Adversarial Learning of General Transformations for Data Augmentation
• Deep Active Learning with Augmentation-based Consistency Estimation
• Data augmentation and image understanding
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• Plexus Convolutional Neural Network (PlexusNet): A novel neural network architecture for histologic image analysis
• Data augmentation with Mobius transformations
• Fast and interpretable classification of small X-ray diffraction datasets using data augmentation and deep neural networks
• Keep Your AI-es on the Road: Tackling Distracted Driver Detection with Convolutional Neural Networks and Targeted Data Augmentation
• Automated Cleanup of the ImageNet Dataset by Model Consensus, Explainability and Confident Learning
• Augmenting the User-Item Graph with Textual Similarity Models
• Domain-Agnostic Clustering with Self-Distillation
• Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification
• Safe Augmentation: Learning Task-Specific Transformations from Data
• Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup
• A Diversity-Enhanced and Constraints-Relaxed Augmentation for Low-Resource Classification
• Fast Jacobian-Vector Product for Deep Networks