Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation

Abstract

Simultaneous speech translation (SimulST) is a challenging task aiming to translate streaming speech before the complete input is observed. A SimulST system generally includes two components: the pre-decision that aggregates the speech information and the policy that decides to read or write. While recent works had proposed various strategies to improve the pre-decision, they mainly adopt the fixed wait-k policy, leaving the adaptive policies rarely explored. This paper proposes to model the adaptive policy by adapting the Continuous Integrate-and-Fire (CIF). Compared with monotonic multihead attention (MMA), our method has the advantage of simpler computation, superior quality at low latency, and better generalization to long utterances. We conduct experiments on the MuST-C V2 dataset and show the effectiveness of our approach.

arXiv commentINTERSPEECH 2022 camera ready

References in corpus (8)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Gaussian Error Linear Units (GELUs)
• Sequence Transduction with Recurrent Neural Networks
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Transformers with convolutional context for ASR
• Can neural machine translation do simultaneous translation?
• Monotonic Multihead Attention
• A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition

Cited by in corpus (2)

• Attention as a Guide for Simultaneous Speech Translation
• Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023