Regularization for Deep Learning: A Taxonomy

Abstract

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• An Overview of Multi-Task Learning in Deep Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles
• Adding noise to the input of a model trained with a regularized objective
• A Bayesian encourages dropout
• Adjusting for Dropout Variance in Batch Normalization and Weight Initialization

Cited by in corpus (11)

• Deep Learning for Human Affect Recognition: Insights and New Developments
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Traditional and Heavy-Tailed Self Regularization in Neural Network Models
• Instance Adaptive Self-Training for Unsupervised Domain Adaptation
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Mixup Regularization for Region Proposal based Object Detectors
• Style transfer-based image synthesis as an efficient regularization technique in deep learning
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
• CoopSubNet: Cooperating Subnetwork for Data-Driven Regularization of Deep Networks under Limited Training Budgets