R-Drop: Regularized Dropout for Neural Networks

Abstract

Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the freedom of the model parameters and complements dropout. Experiments on $5$ widely used deep learning tasks ($18$ datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English$\to$German translation ($30.91$ BLEU) and WMT14 English$\to$French translation ($43.95$ BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub{\url{https://github.com/dropreg/R-Drop}}.

Accepted by NeurIPS 2021

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Language Models are Few-Shot Learners
• Improved Regularization of Convolutional Neural Networks with Cutout
• Pointer Sentinel Mixture Models
• Incorporating BERT into Neural Machine Translation
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Analyzing noise in autoencoders and deep networks
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Dropout with Expectation-linear Regularization
• Not All Attention Is All You Need

Cited by in corpus (1)

• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation