Monotonic Multihead Attention

Abstract

Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attentions heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We also analyze how the latency controls affect the attention span and we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

References in corpus (2)

• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Can neural machine translation do simultaneous translation?

Cited by in corpus (14)

• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• Discriminative Neural Clustering for Speaker Diarisation
• End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
• Does Simultaneous Speech Translation need Simultaneous Models?
• Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection
• Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
• Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff
• Opportunistic Decoding with Timely Correction for Simultaneous Translation
• Infusing Future Information into Monotonic Attention Through Language Models
• Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training
• Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation
• Full-Sentence Models Perform Better in Simultaneous Translation Using the Information Enhanced Decoding Strategy
• Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy