Scheduled DropHead: A Regularization Method for Transformer Models
arXiv:2004.13342
Abstract
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks. In contrast to the conventional dropout mechanisms which randomly drop units or connections, the proposed DropHead is a structured dropout method. It drops entire attention-heads during training and It prevents the multi-head attention model from being dominated by a small portion of attention heads while also reduces the risk of overfitting the training data, thus making use of the multi-head attention mechanism more efficiently. Motivated by recent studies about the learning dynamic of the multi-head attention mechanism, we propose a specific dropout rate schedule to adaptively adjust the dropout rate of DropHead and achieve better regularization effect. Experimental results on both machine translation and text classification benchmark datasets demonstrate the effectiveness of the proposed approach.
arXiv commentEMNLP 2020 Findings
References in corpus (7)
- FractalNet: Ultra-Deep Neural Networks without Residuals
- DropBlock: A regularization method for convolutional networks
- A large annotated corpus for learning natural language inference
- Reducing Transformer Depth on Demand with Structured Dropout
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Weighted Transformer Network for Machine Translation
- DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Cited by in corpus (5)
- Long-Range Transformers for Dynamic Spatiotemporal Forecasting
- Not All Attention Is All You Need
- MHAttnSurv: Multi-Head Attention for Survival Prediction Using Whole-Slide Pathology Images
- Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
- Large Product Key Memory for Pretrained Language Models