Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition

Abstract

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by itself or in various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully self-attentional network for CTC, and show it is tractable and competitive for end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing CTC models and most encoder-decoder models, with character error rates (CERs) of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean, with a fixed architecture and one GPU. Similar improvements hold for WERs after LM decoding. We motivate the architecture for speech, evaluate position and downsampling approaches, and explore how label alphabets (character, phoneme, subword) affect attention heads and performance.

Accepted to ICASSP 2019

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Residual Convolutional CTC Networks for Automatic Speech Recognition

Cited by in corpus (12)

• Self-Attention Transducers for End-to-End Speech Recognition
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Exploring Transformers for Large-Scale Speech Recognition
• Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances
• DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognition
• Weak-Attention Suppression For Transformer Based Speech Recognition
• Research on Modeling Units of Transformer Transducer for Mandarin Speech Recognition
• A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
• FMA-ETA: Estimating Travel Time Entirely Based on FFN With Attention
• Transformer ASR with Contextual Block Processing