Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

Abstract

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.

Submitted to ICASSP 2020

References in corpus (3)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• A Comparative Study on Transformer vs RNN in Speech Applications
• Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learning

Cited by in corpus (31)

• Self-Supervised Speech Representation Learning: A Review
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Unsupervised Speech Recognition
• Curriculum Pre-training for End-to-End Speech Translation
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• Exploring wav2vec 2.0 on speaker verification and language identification
• Self-training and Pre-training are Complementary for Speech Recognition
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Indonesian Automatic Speech Recognition with XLSR-53
• Transformer based unsupervised pre-training for acoustic representation learning
• Towards Semi-Supervised Semantics Understanding from Speech
• CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Understanding Self-Attention of Self-Supervised Audio Transformers
• Transformer with Bidirectional Decoder for Speech Recognition
• DiDiSpeech: A Large Scale Mandarin Speech Corpus
• Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System
• Embodied Self-supervised Learning by Coordinated Sampling and Training
• Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Improved Speech Representations with Multi-Target Autoregressive Predictive Coding
• Similarity Analysis of Self-Supervised Speech Representations
• A Transformer Based Pitch Sequence Autoencoder with MIDI Augmentation
• Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation
• Speech BERT Embedding For Improving Prosody in Neural TTS
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers