Guided contrastive self-supervised pre-training for automatic speech recognition

Abstract

Contrastive Predictive Coding (CPC) is a representation learning method that maximizes the mutual information between intermediate latent representations and the output of a given model. It can be used to effectively initialize the encoder of an Automatic Speech Recognition (ASR) model. We present a novel modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our proposed method maximizes the mutual information between representations from a prior-knowledge model and the output of the model being pre-trained, allowing prior knowledge injection during pre-training. We validate our method on 3 ASR tasks: German, French and English. Our method outperforms CPC pre-training on all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and 15.43% relative on the German, French and English (Librispeech) tasks respectively, compared to training from scratch, while CPC pre-training only brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.

To appear in SLT 2022

References in corpus (8)

• Sequence Transduction with Recurrent Neural Networks
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Unsupervised speech representation learning using WaveNet autoencoders
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset