Dropout as data augmentation

Abstract

Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input space, thereby generating augmented versions of the training data, and we show that training a deterministic network on the augmented samples yields similar results. Finally, we propose a new dropout noise scheme based on our observations and show that it improves dropout results without adding significant computational cost.

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• Theano: new features and speed improvements
• Learning with Pseudo-Ensembles
• Blocks and Fuel: Frameworks for deep learning
• Analyzing noise in autoencoders and deep networks
• Scheduled denoising autoencoders
• Efficient batchwise dropout training using submatrices

Cited by in corpus (21)

• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• Regularization for Deep Learning: A Taxonomy
• Data augmentation instead of explicit regularization
• Applying Deep Learning To Airbnb Search
• Multi-Representational Learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs
• Survey of Dropout Methods for Deep Neural Networks
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Categorical Perception: A Groundwork for Deep Learning
• SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
• Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition
• Interpreting and Boosting Dropout from a Game-Theoretic View
• Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
• Real-Time Steganalysis for Stream Media Based on Multi-channel Convolutional Sliding Windows
• Learning Optimal Data Augmentation Policies via Bayesian Optimization for Image Classification Tasks
• Data augmentation and image understanding
• Stochastic Model Pruning via Weight Dropping Away and Back
• Internal node bagging
• Restrained Generative Adversarial Network against Overfitting in Numeric Data Augmentation
• Manifold Regularized Discriminative Neural Networks
• Generating Diverse Translation from Model Distribution with Dropout
• Only sparsity based loss function for learning representations