A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition

Abstract

Building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, many unsupervised pre-training methods have been proposed. Among these methods, Masked Predictive Coding achieved significant improvements on various speech recognition datasets with BERT-like Masked Reconstruction loss and Transformer backbone. However, many aspects of MPC have not been fully investigated. In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pre-training data speaking style, its extension on streaming model, and how to better transfer learned knowledge from pre-training stage to downstream tasks. Experiments reveled that pre-training data with matching speaking style is more useful on downstream recognition tasks. A unified training objective with APC and MPC provided 8.46% relative error reduction on streaming model trained on HKUST. Also, the combination of target data adaption and layer-wise discriminative training helped the knowledge transfer of MPC, which achieved 3.99% relative error reduction on AISHELL over a strong baseline.

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Effectiveness of self-supervised pre-training for speech recognition
• How to Fine-Tune BERT for Text Classification?
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Learning Robust and Multilingual Speech Representations
• Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learning
• Generative Pre-Training for Speech with Autoregressive Predictive Coding
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks

Cited by in corpus (5)

• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• DiDiSpeech: A Large Scale Mandarin Speech Corpus
• Decoupling recognition and transcription in Mandarin ASR
• Similarity Analysis of Self-Supervised Speech Representations