End-to-end Speech Translation via Cross-modal Progressive Training

Abstract

End-to-end speech translation models have become a new trend in research due to their potential of reducing error propagation. However, these models still suffer from the challenge of data scarcity. How to effectively use unlabeled or other parallel corpora from machine translation is promising but still an open problem. In this paper, we propose Cross Speech-Text Network (XSTNet), an end-to-end model for speech-to-text translation. XSTNet takes both speech and text as input and outputs both transcription and translation text. The model benefits from its three key design aspects: a self-supervised pre-trained sub-network as the audio encoder, a multi-task training objective to exploit additional parallel bilingual text, and a progressive training procedure. We evaluate the performance of XSTNet and baselines on the MuST-C En-X and LibriSpeech En-Fr datasets. In particular, XSTNet achieves state-of-the-art results on all language directions with an average BLEU of 28.8, outperforming the previous best method by 3.2 BLEU. Code, models, cases, and more detailed analysis are available at https://github.com/ReneeYe/XSTNet.

arXiv commentAccepted at InterSpeech2021

References in corpus (11)

• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
• Bridging the Modality Gap for Speech-to-Text Translation
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• Curriculum Pre-training for End-to-End Speech Translation
• Consecutive Decoding for Speech-to-text Translation
• "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation