Regularization for Deep Learning: A Taxonomy

Abstract

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• An Overview of Multi-Task Learning in Deep Neural Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Densely Connected Convolutional Networks
• Multi-Scale Context Aggregation by Dilated Convolutions
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• A guide to convolution arithmetic for deep learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Synthesizing Robust Adversarial Examples
• Semi-Supervised Learning with Ladder Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Adding Gradient Noise Improves Learning for Very Deep Networks
• NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles
• Data-dependent Initializations of Convolutional Neural Networks
• Online Batch Selection for Faster Training of Neural Networks
• Dropout as data augmentation
• Adding noise to the input of a model trained with a regularized objective
• A Bayesian encourages dropout
• Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
• Deep unsupervised learning through spatial contrasting
• Invariant backpropagation: how to train a transformation-invariant neural network

Cited by in corpus (45)

• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Data Augmentation Approaches in Natural Language Processing: A Survey
• Deep Learning for Human Affect Recognition: Insights and New Developments
• Review: Deep Learning in Electron Microscopy
• Data augmentation instead of explicit regularization
• Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting
• An overview of mixing augmentation methods and augmentation strategies
• Logic Explained Networks
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure
• A Systematic Survey of Regularization and Normalization in GANs
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Traditional and Heavy-Tailed Self Regularization in Neural Network Models
• Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder
• Single-frame Regularization for Temporally Stable CNNs
• Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Data
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Instance Adaptive Self-Training for Unsupervised Domain Adaptation
• Particle Identification In Camera Image Sensors Using Computer Vision
• Sparse Transfer Learning via Winning Lottery Tickets
• Multi-structure bone segmentation in pediatric MR images with combined regularization from shape priors and adversarial network
• Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problems
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Improving Robustness of Deep Learning Based Knee MRI Segmentation: Mixup and Adversarial Domain Adaptation
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Multi-Window Data Augmentation Approach for Speech Emotion Recognition
• Post-synaptic potential regularization has potential
• Mixup Regularization for Region Proposal based Object Detectors
• Style transfer-based image synthesis as an efficient regularization technique in deep learning
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks
• Deep Inverse Feature Learning: A Representation Learning of Error
• Enhancing CNN Classification with Lamarckian Memetic Algorithms and Local Search
• Regularized Evolutionary Population-Based Training
• An ETF view of Dropout regularization
• Environment Transfer for Distributed Systems
• Understanding Regularization to Visualize Convolutional Neural Networks
• Muddling Labels for Regularization, a novel approach to generalization
• CoopSubNet: Cooperating Subnetwork for Data-Driven Regularization of Deep Networks under Limited Training Budgets
• Volumization as a Natural Generalization of Weight Decay
• Computational principles of intelligence: learning and reasoning with neural networks
• Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously