Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Abstract

Recent work on end-to-end automatic speech recognition (ASR) has shown that the connectionist temporal classification (CTC) loss can be used to convert acoustics to phone or character sequences. Such systems are used with a dictionary and separately-trained Language Model (LM) to produce word sequences. However, they are not truly end-to-end in the sense of mapping acoustics directly to words without an intermediate phone representation. In this paper, we present the first results employing direct acoustics-to-word CTC models on two well-known public benchmark tasks: Switchboard and CallHome. These models do not require an LM or even a decoder at run-time and hence recognize speech with minimal complexity. However, due to the large number of word output units, CTC word models require orders of magnitude more data to train reliably compared to traditional systems. We present some techniques to mitigate this issue. Our CTC word model achieves a word error rate of 13.0%/18.8% on the Hub5-2000 Switchboard/CallHome test sets without any LM or decoder compared with 9.6%/16.0% for phone-based CTC with a 4-gram LM. We also present rescoring results on CTC word model lattices to quantify the performance benefits of a LM, and contrast the performance of word and phone CTC models.

arXiv commentSubmitted to Interspeech-2017

References in corpus (4)

• cuDNN: Efficient Primitives for Deep Learning
• The Microsoft 2016 Conversational Speech Recognition System
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition

Cited by in corpus (16)

• Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
• Improved training for online end-to-end speech recognition systems
• End-to-end Speech Recognition with Word-based RNN Language Models
• Advancing Acoustic-to-Word CTC Model
• Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition
• Advancing Connectionist Temporal Classification With Attention Modeling
• Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR
• Recent Progresses in Deep Learning based Acoustic Models (Updated)
• A GPU-based WFST Decoder with Exact Lattice Generation
• Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition
• Learning Shared Encoding Representation for End-to-End Speech Recognition Models
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers
• Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMs
• Acoustic-to-Word Models with Conversational Context Information
• Training a code-switching language model with monolingual data
• LSTM Acoustic Models Learn to Align and Pronounce with Graphemes