Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation

Abstract

Predicting the altered acoustic frames is an effective way of self-supervised learning for speech representation. However, it is challenging to prevent the pretrained model from overfitting. In this paper, we proposed to introduce two dropout regularization methods into the pretraining of transformer encoder: (1) attention dropout, (2) layer dropout. Both of the two dropout methods encourage the model to utilize global speech information, and avoid just copying local spectrum features when reconstructing the masked frames. We evaluated the proposed methods on phoneme classification and speaker recognition tasks. The experiments demonstrate that our dropout approaches achieve competitive results, and improve the performance of classification accuracy on downstream tasks.

arXiv commentwill be presented in INTERSPEECH 2021

References in corpus (9)

• Improving neural networks by preventing co-adaptation of feature detectors
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
• Attention-based Dropout Layer for Weakly Supervised Object Localization
• Unsupervised pre-training for sequence to sequence speech recognition
• Cascade Attentive Dropout for Weakly Supervised Object Detection