End-to-End Speech Translation with Knowledge Distillation

Abstract

End-to-end speech translation (ST), which directly translates from source language speech into target language text, has attracted intensive attentions in recent years. Compared to conventional pipeline systems, end-to-end ST models have advantages of lower latency, smaller model size and less error propagation. However, the combination of speech recognition and text translation in one model is more difficult than each of these two tasks. In this paper, we propose a knowledge distillation approach to improve ST model by transferring the knowledge from text translation model. Specifically, we first train a text translation model, regarded as a teacher model, and then ST model is trained to learn output probabilities from teacher model through knowledge distillation. Experiments on English- French Augmented LibriSpeech and English-Chinese TED corpus show that end-to-end ST is possible to implement on both similar and dissimilar language pairs. In addition, with the instruction of teacher model, end-to-end ST model can gain significant improvements by over 3.5 BLEU points.

Submitted to Interspeech 2019

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Large scale distributed neural network training through online distillation
• Multilingual Neural Machine Translation with Knowledge Distillation
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Ensemble Distillation for Neural Machine Translation

Cited by in corpus (14)

• Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• On Using SpecAugment for End-to-End Speech Translation
• MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation
• Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection
• Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• NeurST: Neural Speech Translation Toolkit
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• Multilingual End-to-End Speech Translation
• Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• Neural Machine Translation: A Review and Survey
• Robust Machine Translation with Domain Sensitive Pseudo-Sources: Baidu-OSU WMT19 MT Robustness Shared Task System Report
• The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021