Transformer-based Acoustic Modeling for Hybrid Speech Recognition

Abstract

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.

arXiv commentto appear in ICASSP 2020

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Gaussian Error Linear Units (GELUs)
• A Comparative Study on Transformer vs RNN in Speech Applications
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Transformers with convolutional context for ASR
• RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
• Feedforward Sequential Memory Neural Networks without Recurrent Feedback
• From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition

Cited by in corpus (57)

• Conformer: Convolution-augmented Transformer for Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Effectiveness of self-supervised pre-training for speech recognition
• SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention
• Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition
• MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction
• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• Iterative Pseudo-Labeling for Speech Recognition
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• Semantic Mask for Transformer based End-to-End Speech Recognition
• Low Latency End-to-End Streaming Speech Recognition with a Scout Network
• TEMGNet: Deep Transformer-based Decoding of Upperlimb sEMG for Hand Gestures Recognition
• Bayesian Learning for Deep Neural Network Adaptation
• HiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Multilingual Graphemic Hybrid ASR with Massive Data Augmentation
• Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition
• Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition
• Efficient Training of Neural Transducer for Speech Recognition
• Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• FluentNet: End-to-End Detection of Speech Disfluency with Deep Learning
• Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• Weak-Attention Suppression For Transformer Based Speech Recognition
• On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers
• Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers
• Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition
• Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning
• ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition
• A Better and Faster End-to-End Model for Streaming ASR
• Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
• Contrastive Semi-supervised Learning for ASR
• Adversarial Data Augmentation for Disordered Speech Recognition
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition
• TransMask: A Compact and Fast Speech Separation Model Based on Transformer
• When Can Self-Attention Be Replaced by Feed Forward Layers?
• G2G: TTS-Driven Pronunciation Learning for Graphemic Hybrid ASR
• Accelerator-Aware Training for Transducer-Based Speech Recognition
• Cross-attention conformer for context modeling in speech enhancement for ASR
• Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications
• End-to-End Multi-Channel Transformer for Speech Recognition
• Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR
• Effective Decoder Masking for Transformer Based End-to-End Speech Recognition
• Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios
• Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory
• Far-Field Automatic Speech Recognition
• Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASR
• Multi-Channel Transformer Transducer for Speech Recognition
• Explaining the Attention Mechanism of End-to-End Speech Recognition Using Decision Trees
• Population Based Training for Data Augmentation and Regularization in Speech Recognition
• Nana-HDR: A Non-attentive Non-autoregressive Hybrid Model for TTS
• Transformer Based Deliberation for Two-Pass Speech Recognition