DropDim: A Regularization Method for Transformer Networks
arXiv:2304.10321 · doi:10.1109/LSP.2022.3140693
Abstract
We introduceDropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is a key component of the transformer. In contrast to the general dropout method, which randomly drops neurons, DropDim drops part of the embedding dimensions. In this way, the semantic information can be completely discarded. Thus, the excessive coadapting between different embedding dimensions can be broken, and the self-attention is forced to encode meaningful featureswith a certain number of embedding dimensions erased. Experiments on a wide range of tasks executed on the MUST-C English-Germany dataset show that DropDim can effectively improve model performance, reduce over-fitting, and show complementary effects with other regularization methods. When combined with label smoothing, the WER can be reduced from 19.1% to 15.1% on the ASR task, and the BLEU value can be increased from26.90 to 28.38 on the MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase by 1.86 BLEU points compared to the strong baseline.
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
- Data Noising as Smoothing in Neural Network Language Models
- A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
- Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation
- DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks