Not All Attention Is All You Need
arXiv:2104.04692
Abstract
Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.
References in corpus (14)
- DARTS: Differentiable Architecture Search
- Linformer: Self-Attention with Linear Complexity
- Layer Normalization
- Variational Dropout Sparsifies Deep Neural Networks
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
- EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
- Auto-Sizing Neural Networks: With Applications to n-gram Language Models
- Alleviating the Inequality of Attention Heads for Neural Machine Translation
- Scheduled DropHead: A Regularization Method for Transformer Models
- Code Summarization with Structure-induced Transformer