Dropout Training as Adaptive Regularization

Abstract

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

11 pages. Advances in Neural Information Processing Systems (NIPS), 2013

References in corpus (2)

• Improving neural networks by preventing co-adaptation of feature detectors
• Adding noise to the input of a model trained with a regularized objective

Cited by in corpus (91)

• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Learning with Pseudo-Ensembles
• Towards Dropout Training for Convolutional Neural Networks
• On Complex Valued Convolutional Neural Networks
• Data augmentation instead of explicit regularization
• Physics-Driven Regularization of Deep Neural Networks for Enhanced Engineering Design and Analysis
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Dropout as a Bayesian Approximation: Appendix
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Batch Virtual Adversarial Training for Graph Convolutional Networks
• Dynamic Prediction of ICU Mortality Risk Using Domain Adaptation
• Survey of Dropout Methods for Deep Neural Networks
• A Selective Overview of Deep Learning
• Dimensionality compression and expansion in Deep Neural Networks
• Market Trend Prediction using Sentiment Analysis: Lessons Learned and Paths Forward
• Explicit Regularisation in Gaussian Noise Injections
• A Bayesian encourages dropout
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
• Online Linear Optimization via Smoothing
• The Implicit and Explicit Regularization Effects of Dropout
• CASTLE: Regularization via Auxiliary Causal Graph Discovery
• Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation
• Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• How Does Mixup Help With Robustness and Generalization?
• Altitude Training: Strong Bounds for Single-Layer Dropout
• Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
• On the Inductive Bias of Dropout
• Half-CNN: A General Framework for Whole-Image Regression
• The Implicit Regularization of Ordinary Least Squares Ensembles
• Telugu OCR Framework using Deep Learning
• On Convergence and Generalization of Dropout Training
• Dropout Training for Support Vector Machines
• Dropout: Explicit Forms and Capacity Control
• Semi-Supervised Brain Lesion Segmentation with an Adapted Mean Teacher Model
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• On Mixup Regularization
• On Dropout and Nuclear Norm Regularization
• Machine Learning's Dropout Training is Distributionally Robust Optimal
• Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution
• Faster Neural Network Training with Approximate Tensor Operations
• Orthogonal Deep Neural Networks
• Dropout Training of Matrix Factorization and Autoencoder for Link Prediction in Sparse Graphs
• Interpreting and Boosting Dropout from a Game-Theoretic View
• Robust Regression for Safe Exploration in Control
• Sparseout: Controlling Sparsity in Deep Networks
• Consistent Sparse Deep Learning: Theory and Computation
• Almost Sure Convergence of Dropout Algorithms for Neural Networks
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• PANDA: AdaPtive Noisy Data Augmentation for Regularization of Undirected Graphical Models
• Max-Affine Spline Insights into Deep Generative Networks
• Deep Online Convex Optimization with Gated Games
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error
• Post-synaptic potential regularization has potential
• Implicit Regularization and Convergence for Weight Normalization
• Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Stochastic Sparse Subspace Clustering
• Learning Dynamics of Linear Denoising Autoencoders
• Distributionally Robust Parametric Maximum Likelihood Estimation
• Xiaomingbot: A Multilingual Robot News Reporter
• Partial Graph Reasoning for Neural Network Regularization
• Data augmentation and image understanding
• Dropout as a Regularizer of Interaction Effects
• Structured Dropout Variational Inference for Bayesian Neural Networks
• DropFilter: A Novel Regularization Method for Learning Convolutional Neural Networks
• An ETF view of Dropout regularization
• Surprising properties of dropout in deep networks
• Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines
• On approximating dropout noise injection
• An Analysis of Dropout for Matrix Factorization
• Matrix Sketching for Secure Collaborative Machine Learning
• Principled learning method for Wasserstein distributionally robust optimization with local perturbations
• Why Do Better Loss Functions Lead to Less Transferable Features?
• How Data Augmentation affects Optimization for Linear Regression
• Lockout: Sparse Regularization of Neural Networks
• Neural Gaussian Mirror for Controlled Feature Selection in Neural Networks
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
• Designing Machine Learning Surrogates using Outputs of Molecular Dynamics Simulations as Soft Labels
• Efficient Modelling Across Time of Human Actions and Interactions
• Regularization in ResNet with Stochastic Depth
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• Deep Autoencoders: From Understanding to Generalization Guarantees
• Extracting representations of cognition across neuroimaging studies improves brain decoding
• Medi-Care AI: Predicting Medications From Billing Codes via Robust Recurrent Neural Networks
• Inherent Noise in Gradient Based Methods
• Shakeout: A New Approach to Regularized Deep Neural Network Training
• Manifold Regularized Discriminative Neural Networks
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions