Multilingual End-to-End Speech Translation

Abstract

In this paper, we propose a simple yet effective framework for multilingual end-to-end speech translation (ST), in which speech utterances in source languages are directly translated to the desired target languages with a universal sequence-to-sequence architecture. While multilingual models have shown to be useful for automatic speech recognition (ASR) and machine translation (MT), this is the first time they are applied to the end-to-end ST problem. We show the effectiveness of multilingual end-to-end ST in two scenarios: one-to-many and many-to-many translations with publicly available data. We experimentally confirm that multilingual end-to-end ST models significantly outperform bilingual ones in both scenarios. The generalization of multilingual training is also evaluated in a transfer learning scenario to a very low-resource language pair. All of our codes and the database are publicly available to encourage further research in this emergent multilingual ST topic.

arXiv commentAccepted to ASRU 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Recurrent Neural Network Regularization
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Effective Strategies in Zero-Shot Neural Machine Translation
• End-to-End Speech Translation with Knowledge Distillation
• Unsupervised Word Segmentation from Speech with Attention
• Fine-tuning on Clean Data for End-to-End Speech Translation: FBK @ IWSLT 2018

Cited by in corpus (3)

• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
• Self-Supervised Representations Improve End-to-End Speech Translation
• Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation