Improved training of end-to-end attention models for speech recognition

Abstract

Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition. In this work, we show that such models can achieve competitive results on the Switchboard 300h and LibriSpeech 1000h tasks. In particular, we report the state-of-the-art word error rates (WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets of LibriSpeech. We introduce a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance. In some experiments, we also use an auxiliary CTC loss function to help the convergence. In addition, we train long short-term memory (LSTM) language models on subword units. By shallow fusion, we report up to 27% relative improvements in WER over the attention baseline without a language model.

submitted to Interspeech 2018

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Multiple Object Recognition with Visual Attention
• On Using Monolingual Corpora in Neural Machine Translation
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Advances in All-Neural Speech Recognition
• The CAPIO 2017 Conversational Speech Recognition System
• RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
• Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
• Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
• Letter-Based Speech Recognition with Gated ConvNets
• Improving End-to-End Speech Recognition with Policy Learning

Cited by in corpus (76)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Self-Training for End-to-End Speech Recognition
• Language Modeling with Deep Transformers
• Transformers with convolutional context for ASR
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Fully Convolutional Speech Recognition
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Jasper: An End-to-End Convolutional Neural Acoustic Model
• A New Training Pipeline for an Improved Neural Transducer
• Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
• Hierarchical Multitask Learning for CTC-based Speech Recognition
• Letter-Based Speech Recognition with Gated ConvNets
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Towards Understanding Label Smoothing
• Who Needs Words? Lexicon-Free Speech Recognition
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• On Using SpecAugment for End-to-End Speech Translation
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription
• On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition
• A novel pyramidal-FSMN architecture with lattice-free MMI for speech recognition
• An improved hybrid CTC-Attention model for speech recognition
• Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• An investigation of phone-based subword units for end-to-end speech recognition
• CAT: CRF-based ASR Toolkit
• end-to-end training of a large vocabulary end-to-end speech recognition system
• From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition
• IMS-Speech: A Speech to Text Tool
• A study of latent monotonic attention variants
• Gated Recurrent Context: Softmax-free Attention for Online Encoder-Decoder Speech Recognition
• End-to-End Whisper to Natural Speech Conversion using Modified Transformer Network
• End-to-End ASR for Code-switched Hindi-English Speech
• CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
• Transfer learning of language-independent end-to-end ASR with language model fusion
• Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances
• Automatic Speech Recognition Benchmark for Air-Traffic Communications
• A systematic comparison of grapheme-based vs. phoneme-based label units for encoder-decoder-attention models
• Optimal Completion Distillation for Sequence Learning
• Equivalence of Segmental and Neural Transducer Modeling: A Proof of Concept
• Cross-Attention End-to-End ASR for Two-Party Conversations
• Effective Data Augmentation with Multi-Domain Learning GANs
• Data Techniques For Online End-to-end Speech Recognition
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscripts
• Insertion-Based Modeling for End-to-End Automatic Speech Recognition
• power-law nonlinearity with maximally uniform distribution criterion for improved neural network training in automatic speech recognition
• Attention-based sequence-to-sequence model for speech recognition: development of state-of-the-art system on LibriSpeech and its application to non-native English
• KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition
• Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition
• Text-Independent Speaker Verification with Dual Attention Network
• A spelling correction model for end-to-end speech recognition
• End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection
• G2G: TTS-Driven Pronunciation Learning for Graphemic Hybrid ASR
• Learning Shared Encoding Representation for End-to-End Speech Recognition Models
• Improving Reverberant Speech Separation with Multi-stage Training and Curriculum Learning
• Deep Shallow Fusion for RNN-T Personalization
• CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency
• Monolingual Data Selection Analysis for English-Mandarin Hybrid Code-switching Speech Recognition
• Transformer ASR with Contextual Block Processing
• Attention-based ASR with Lightweight and Dynamic Convolutions
• Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models
• Vectorization of hypotheses and speech for faster beam search in encoder decoder-based speech recognition
• Audio-Visual Decision Fusion for WFST-based and seq2seq Models
• Cascaded CNN-resBiLSTM-CTC: An End-to-End Acoustic Model For Speech Recognition
• End-To-End Speech Recognition Using A High Rank LSTM-CTC Based Model
• Acoustic-to-Word Models with Conversational Context Information
• Quantifying and Maximizing the Benefits of Back-End Noise Adaption on Attention-Based Speech Recognition Models
• Small energy masking for improved neural network training for end-to-end speech recognition
• Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling
• HMM-Free Encoder Pre-Training for Streaming RNN Transducer
• User-friendly automatic transcription of low-resource languages: Plugging ESPnet into Elpis
• Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition
• Far-Field Automatic Speech Recognition
• Attention based on-device streaming speech recognition with large speech corpus