Weak-Attention Suppression For Transformer Based Speech Recognition
arXiv:2005.09137
Abstract
Transformers, originally proposed for natural language processing (NLP) tasks, have recently achieved great success in automatic speech recognition (ASR). However, adjacent acoustic units (i.e., frames) are highly correlated, and long-distance dependencies between them are weak, unlike text units. It suggests that ASR will likely benefit from sparse and localized attention. In this paper, we propose Weak-Attention Suppression (WAS), a method that dynamically induces sparsity in attention probabilities. We demonstrate that WAS leads to consistent Word Error Rate (WER) improvement over strong transformer baselines. On the widely used LibriSpeech benchmark, our proposed method reduced WER by 10%$ on test-clean and 5% on test-other for streamable transformers, resulting in a new state-of-the-art among streaming models. Further analysis shows that WAS learns to suppress attention of non-critical and redundant continuous acoustic frames, and is more likely to suppress past frames rather than future ones. It indicates the importance of lookahead in attention-based ASR models.
submitted to interspeech 2020
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- A Comparative Study on Transformer vs RNN in Speech Applications
- Reformer: The Efficient Transformer
- Transformer-based Acoustic Modeling for Hybrid Speech Recognition
- Transformers with convolutional context for ASR
- Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
- Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
- Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
- Adaptively Sparse Transformers