Towards better decoding and language model integration in sequence to sequence models

Abstract

The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion. In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters. We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used. We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Asynchronous Methods for Deep Reinforcement Learning
• Effective Approaches to Attention-based Neural Machine Translation
• On Using Monolingual Corpora in Neural Machine Translation
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Modeling Coverage for Neural Machine Translation
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Learning Online Alignments with Continuous Rewards Policy Gradient
• Very Deep Convolutional Networks for End-to-End Speech Recognition

Cited by in corpus (44)

• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Hierarchical Neural Story Generation
• Exploring Neural Transducers for End-to-End Speech Recognition
• Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Sequence-to-Sequence Models Can Directly Translate Foreign Speech
• Towards End-to-End Code-Switching Speech Recognition
• Recent advances in conversational NLP : Towards the standardization of Chatbot building
• Letter-Based Speech Recognition with Gated ConvNets
• Monotonic Chunkwise Attention
• QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
• Unsupervised pre-training for sequence to sequence speech recognition
• Multi-encoder multi-resolution framework for end-to-end speech recognition
• End-to-end Speech Recognition with Word-based RNN Language Models
• Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
• Improving End-to-End Speech Recognition with Policy Learning
• Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion
• Improved Chord Recognition by Combining Duration and Harmonic Language Models
• Simplified End-to-End MMI Training and Voting for ASR
• An Online Attention-based Model for Speech Recognition
• A comparable study of modeling units for end-to-end Mandarin speech recognition
• Why are Sequence-to-Sequence Models So Dull? Understanding the Low-Diversity Problem of Chatbots
• Fusion Recurrent Neural Network
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Multi-Stream End-to-End Speech Recognition
• Simple Unsupervised Summarization by Contextual Matching
• Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement
• Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR
• Data Troubles in Sentence Level Confidence Estimation for Machine Translation
• Attention-Based End-to-End Speech Recognition on Voice Search
• Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-based LVCSR
• Automatic Chord Recognition with Higher-Order Harmonic Language Modelling
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer
• Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
• Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition
• Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
• Audio-Visual Decision Fusion for WFST-based and seq2seq Models
• CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition
• Chi-square Loss for Softmax: an Echo of Neural Network Structure
• Character-Aware Attention-Based End-to-End Speech Recognition
• Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
• Normalized Label Distribution: Towards Learning Calibrated, Adaptable and Efficient Activation Maps