Transformer-Transducer: End-to-End Speech Recognition with Self-Attention

Abstract

We explore options to use Transformer networks in neural transducer for end-to-end speech recognition. Transformer networks use self-attention for sequence modeling and comes with advantages in parallel computation and capturing contexts. We propose 1) using VGGNet with causal convolution to incorporate positional information and reduce frame rate for efficient inference 2) using truncated self-attention to enable streaming for Transformer and reduce computational complexity. All experiments are conducted on the public LibriSpeech corpus. The proposed Transformer-Transducer outperforms neural transducer with LSTM/BLSTM networks and achieved word error rates of 6.37 % on the test-clean set and 15.30 % on the test-other set, while remaining streamable, compact with 45.7M parameters for the entire system, and computationally efficient with complexity of O(T), where T is input sequence length.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Convolutional Sequence to Sequence Learning
• Sequence Transduction with Recurrent Neural Networks
• Achieving Human Parity in Conversational Speech Recognition
• Transformers with convolutional context for ASR

Cited by in corpus (36)

• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• A New Training Pipeline for an Improved Neural Transducer
• Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription
• Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Attention-based Transducer for Online Speech Recognition
• FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
• Weak-Attention Suppression For Transformer Based Speech Recognition
• Transformer-based Arabic Dialect Identification
• Efficient End-to-End Speech Recognition Using Performers in Conformers
• Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition
• Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-End Speech Recognition
• Learning Word-Level Confidence For Subword End-to-End ASR
• Unidirectional Memory-Self-Attention Transducer for Online Speech Recognition
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
• On Addressing Practical Challenges for RNN-Transducer
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
• Cross-attention conformer for context modeling in speech enhancement for ASR
• Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios
• Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory
• A Light-weight contextual spelling correction model for customizing transducer-based speech recognition systems
• Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency
• FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization
• CarneliNet: Neural Mixture Model for Automatic Speech Recognition
• Multi-mode Transformer Transducer with Stochastic Future Context
• Multi-Channel Transformer Transducer for Speech Recognition
• Context-Aware Transformer Transducer for Speech Recognition
• Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition
• Transformer Based Deliberation for Two-Pass Speech Recognition