Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
arXiv:2110.15729
Abstract
Simultaneous translation systems start producing the output while processing the partial source sentence in the incoming input stream. These systems need to decide when to read more input and when to write the output. These decisions depend on the structure of source/target language and the information contained in the partial input sequence. Hence, read/write decision policy remains the same across different input modalities, i.e., speech and text. This motivates us to leverage the text transcripts corresponding to the speech input for improving simultaneous speech-to-text translation (SimulST). We propose Decision Attentive Regularization (DAR) to improve the decision policy of SimulST systems by using the simultaneous text-to-text translation (SimulMT) task. We also extend several techniques from the offline speech translation domain to explore the role of SimulMT task in improving SimulST performance. Overall, we achieve 34.66% / 4.5 BLEU improvement over the baseline model across different latency regimes for the MuST-C English-German (EnDe) SimulST task.
arXiv comment5 pages, 3 figures, 1 table
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Online and Linear-Time Attention by Enforcing Monotonic Alignments
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Monotonic Multihead Attention
- Monotonic Chunkwise Attention
- STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework
- Multi-Modal Data Augmentation for End-to-End ASR
- Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task