R-Drop: Regularized Dropout for Neural Networks

Abstract

Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the freedom of the model parameters and complements dropout. Experiments on $5$ widely used deep learning tasks ($18$ datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English$\to$German translation ($30.91$ BLEU) and WMT14 English$\to$French translation ($43.95$ BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub{\url{https://github.com/dropreg/R-Drop}}.

Accepted by NeurIPS 2021

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Language Models are Few-Shot Learners
• Teaching Machines to Read and Comprehend
• Improved Regularization of Convolutional Neural Networks with Cutout
• Pointer Sentinel Mixture Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Variational Dropout Sparsifies Deep Neural Networks
• Towards Dropout Training for Convolutional Neural Networks
• Incorporating BERT into Neural Machine Translation
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Very Deep Transformers for Neural Machine Translation
• Analyzing noise in autoencoders and deep networks
• Survey of Dropout Methods for Deep Neural Networks
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
• Dropout with Expectation-linear Regularization
• Not All Attention Is All You Need

Cited by in corpus (9)

• Iterative pseudo-labeling based adaptive copy-paste supervision for semi-supervised tumor segmentation
• Segmenting Medical Images with Limited Data
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• FABind+: Enhancing Molecular Docking through Improved Pocket Prediction and Pose Generation
• UNGT: Ultrasound Nasogastric Tube Dataset for Medical Image Analysis
• Extractive Text Summarization Using Generalized Additive Models with Interactions for Sentence Selection
• Rega-Net:Retina Gabor Attention for Deep Convolutional Neural Networks
• Multi-Domain Transformer-Based Counterfactual Augmentation for Earnings Call Analysis
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation