Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR

Abstract

Recently, self-supervised learning (SSL) from unlabelled speech data has gained increased attention in the automatic speech recognition (ASR) community. Typical SSL methods include autoregressive predictive coding (APC), Wav2vec2.0, and hidden unit BERT (HuBERT). However, SSL models are biased to the pretraining data. When SSL models are finetuned with data from another domain, domain shifting occurs and might cause limited knowledge transfer for downstream tasks. In this paper, we propose a novel framework, domain responsible adaptation and finetuning (DRAFT), to reduce domain shifting in pretrained speech models, and evaluate it for a causal and non-causal transformer. For the causal transformer, an extension of APC (E-APC) is proposed to learn richer information from unlabelled data by using multiple temporally-shifted sequences to perform prediction. For the non-causal transformer, various solutions for using the bidirectional APC (Bi-APC) are investigated. In addition, the DRAFT framework is examined for Wav2vec2.0 and HuBERT methods, which use non-causal transformers as the backbone. The experiments are conducted on child ASR (using the OGI and MyST databases) using SSL models trained with unlabelled adult speech data from Librispeech. The relative WER improvements of up to 19.7% on the two child tasks are observed when compared to the pretrained models without adaptation. With the proposed methods (E-APC and DRAFT), the relative WER improvements are even larger (30% and 19% on the OGI and MyST data, respectively) when compared to the models without using pretraining methods.

arXiv commentPublished in IEEE Journal of Selected Topics in Signal Processing, ICASSP Journal Poster Presentation

References in corpus (23)

• Representation Learning with Contrastive Predictive Coding
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Effectiveness of self-supervised pre-training for speech recognition
• Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
• Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings
• Exploiting Adapters for Cross-lingual Low-resource Speech Recognition
• An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
• Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition
• Incremental Layer-wise Self-Supervised Learning for Efficient Speech Domain Adaptation On Device
• Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System
• Don't speak too fast: The impact of data bias on self-supervised speech models

Cited by in corpus (4)

• Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition
• UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models