DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
arXiv:1907.11065
Abstract
Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.
References in corpus (8)
- Improved Regularization of Convolutional Neural Networks with Cutout
- A Structured Self-attentive Sentence Embedding
- DropBlock: A regularization method for convolutional networks
- Shake-Shake regularization
- Multi-Task Cross-Lingual Sequence Tagging from Scratch
- ShakeDrop Regularization for Deep Residual Learning
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Recurrent Dropout without Memory Loss
Cited by in corpus (7)
- Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
- R-Drop: Regularized Dropout for Neural Networks
- Spatial Temporal Transformer Network for Skeleton-based Action Recognition
- DropDim: A Regularization Method for Transformer Networks
- Scheduled DropHead: A Regularization Method for Transformer Models
- Large Language Models -- the Future of Fundamental Physics?
- Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation