Leveraging native language information for improved accented speech recognition

Abstract

Recognition of accented speech is a long-standing challenge for automatic speech recognition (ASR) systems, given the increasing worldwide population of bi-lingual speakers with English as their second language. If we consider foreign-accented speech as an interpolation of the native language (L1) and English (L2), using a model that can simultaneously address both languages would perform better at the acoustic level for accented speech. In this study, we explore how an end-to-end recurrent neural network (RNN) trained system with English and native languages (Spanish and Indian languages) could leverage data of native languages to improve performance for accented English speech. To this end, we examine pre-training with native languages, as well as multi-task learning (MTL) in which the main task is trained with native English and the secondary task is trained with Spanish or Indian Languages. We show that the proposed MTL model performs better than the pre-training approach and outperforms a baseline model trained simply with English data. We suggest a new setting for MTL in which the secondary task is trained with both English and the native language, using the same output set. This proposed scenario yields better performance with +11.95% and +17.55% character error rate gains over baseline for Hispanic and Indian accents, respectively.

Accepted at Interspeech 2018

References in corpus (4)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Achieving Human Parity in Conversational Speech Recognition
• Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model

Cited by in corpus (5)

• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Accented Speech Recognition: A Survey
• E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition
• Advancing Multi-Accented LSTM-CTC Speech Recognition using a Domain Specific Student-Teacher Learning Paradigm
• Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech