Learning Shared Semantic Space for Speech-to-Text Translation
arXiv:2105.03095
Abstract
Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In observation of this obstacle, we propose to bridge this representation gap with Chimera. By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN-DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demonstrate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for augmenting training resources across modalities. Code, data, and resources are available at https://github.com/Glaciohound/Chimera-ST.
arXiv comment9 pages, 5 figures, Accepted by Findings of ACL 2021
References in corpus (13)
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
- Bridging the Modality Gap for Speech-to-Text Translation
- Multilingual NMT with a language-independent attention bridge
- Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
- On Using SpecAugment for End-to-End Speech Translation
- A neural interlingua for multilingual machine translation
- Towards Robust Neural Machine Translation
- The USTC-NEL Speech Translation system at IWSLT 2018
- Consecutive Decoding for Speech-to-text Translation
- "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
- Lattice Transformer for Speech Translation