Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition

Abstract

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by itself or in various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully self-attentional network for CTC, and show it is tractable and competitive for end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing CTC models and most encoder-decoder models, with character error rates (CERs) of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean, with a fixed architecture and one GPU. Similar improvements hold for WERs after LM decoding. We motivate the architecture for speech, evaluate position and downsampling approaches, and explore how label alphabets (character, phoneme, subword) affect attention heads and performance.

Accepted to ICASSP 2019

References in corpus (10)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Improved training of end-to-end attention models for speech recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Residual Convolutional CTC Networks for Automatic Speech Recognition
• Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages
• Letter-Based Speech Recognition with Gated ConvNets

Cited by in corpus (31)

• A General Survey on Attention Mechanisms in Deep Learning
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Language Modeling with Deep Transformers
• Transformers with convolutional context for ASR
• Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning Based Methods
• Self-Attention Transducers for End-to-End Speech Recognition
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Non-autoregressive Transformer-based End-to-end ASR using BERT
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Exploring Transformers for Large-Scale Speech Recognition
• Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks
• Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition
• A Hardware-Oriented and Memory-Efficient Method for CTC Decoding
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
• Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances
• DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognition
• Weak-Attention Suppression For Transformer Based Speech Recognition
• Transformer-based Arabic Dialect Identification
• Research on Modeling Units of Transformer Transducer for Mandarin Speech Recognition
• FMA-ETA: Estimating Travel Time Entirely Based on FFN With Attention
• A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
• Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition
• Insertion-Based Modeling for End-to-End Automatic Speech Recognition
• Unidirectional Memory-Self-Attention Transducer for Online Speech Recognition
• Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-End Speech Recognition
• Transformer ASR with Contextual Block Processing
• Streaming Transformer ASR with Blockwise Synchronous Beam Search
• Gaussian Kernelized Self-Attention for Long Sequence Data and Its Application to CTC-based Speech Recognition