Thinking Slow about Latency Evaluation for Simultaneous Machine Translation

Abstract

Simultaneous machine translation attempts to translate a source sentence before it is finished being spoken, with applications to translation of spoken language for live streaming and conversation. Since simultaneous systems trade quality to reduce latency, having an effective and interpretable latency metric is crucial. We introduce a variant of the recently proposed Average Lagging (AL) metric, which we call Differentiable Average Lagging (DAL). It distinguishes itself by being differentiable and internally consistent to its underlying mathematical model.

Cited by in corpus (6)

• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• SimulEval: An Evaluation Toolkit for Simultaneous Translation
• Opportunistic Decoding with Timely Correction for Simultaneous Translation
• A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation
• Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation
• Stream-level Latency Evaluation for Simultaneous Machine Translation