Online and Linear-Time Attention by Enforcing Monotonic Alignments
arXiv:1704.00784
Abstract
Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.
arXiv commentICML camera-ready version; 10 pages + 9 page appendix
References in corpus (13)
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Categorical Reparameterization with Gumbel-Softmax
- Attention-Based Models for Speech Recognition
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Effective Approaches to Attention-based Neural Machine Translation
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Neural Turing Machines
- Towards better decoding and language model integration in sequence to sequence models
- A Neural Transducer
Cited by in corpus (62)
- Improved training of end-to-end attention models for speech recognition
- A Survey on Neural Speech Synthesis
- Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
- WT5?! Training Text-to-Text Models to Explain their Predictions
- Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
- Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
- Monotonic Multihead Attention
- Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture
- Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
- Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition
- End-to-End Adversarial Text-to-Speech
- Multimodal Speech Emotion Recognition using Cross Attention with Aligned Audio and Text
- Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
- AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN
- Online Automatic Speech Recognition with Listen, Attend and Spell Model
- On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
- A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition
- WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit
- Decoding-History-Based Adaptive Control of Attention for Neural Machine Translation
- Multi-head Monotonic Chunkwise Attention For Online Speech Recognition
- Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition
- End-to-End Simultaneous Speech Translation with Differentiable Segmentation
- Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
- Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning
- Universal ASR: Unifying Streaming and Non-Streaming ASR Using a Single Encoder-Decoder Model
- Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
- Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection
- Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
- Class LM and word mapping for contextual biasing in End-to-End ASR
- A study of latent monotonic attention variants
- Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS
- An Online Attention-based Model for Speech Recognition
- Morphological Inflection Generation with Hard Monotonic Attention
- Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances
- Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments
- AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms
- Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
- Multi Task Deep Morphological Analyzer: Context Aware Joint Morphological Tagging and Lemma Prediction
- MoBoAligner: a Neural Alignment Model for Non-autoregressive TTS with Monotonic Boundary Search
- Memory-Augmented Neural Networks for Machine Translation
- Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hopping
- Simple Unsupervised Summarization by Contextual Matching
- Multi-Stream End-to-End Speech Recognition
- Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin
- On-device neural speech synthesis
- Neural Machine Translation: A Review and Survey
- A Better and Faster End-to-End Model for Streaming ASR
- Tackling Sequence to Sequence Mapping Problems with Neural Networks
- Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model
- End-to-end Speech Recognition with Adaptive Computation Steps
- Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models
- Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR
- Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesis
- FeatherTTS: Robust and Efficient attention based Neural TTS
- Extremely Low Footprint End-to-End ASR System for Smart Device
- A comparison of streaming models and data augmentation methods for robust speech recognition
- Multi-mode Transformer Transducer with Stochastic Future Context
- Attention based on-device streaming speech recognition with large speech corpus
- Quantum Statistics-Inspired Neural Attention
- Can DNNs Learn to Lipread Full Sentences?
- Mutually-Constrained Monotonic Multihead Attention for Online ASR
- Sequence-to-Sequence Learning with Latent Neural Grammars