Bridging the Modality Gap for Speech-to-Text Translation

Abstract

End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously, which ignores the speech-and-text modality differences and makes the encoder overloaded, leading to great difficulty in learning such a model. To address these issues, we propose a Speech-to-Text Adaptation for Speech Translation (STAST) model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text. Specifically, we decouple the speech translation encoder into three parts and introduce a shrink mechanism to match the length of speech representation with that of the corresponding text transcription. To obtain better semantic representation, we completely integrate a text-based translation model into the STAST so that two tasks can be trained in the same latent space. Furthermore, we introduce a cross-modal adaptation method to close the distance between speech and text representation. Experimental results on English-French and English-German speech translation corpora have shown that our model significantly outperforms strong baselines, and achieves the new state-of-the-art performance.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Large scale distributed neural network training through online distillation
• Multilingual Neural Machine Translation with Knowledge Distillation
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Ensemble Distillation for Neural Machine Translation
• On Using SpecAugment for End-to-End Speech Translation

Cited by in corpus (9)

• Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
• End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• Does Simultaneous Speech Translation need Simultaneous Models?
• End-to-end Speech Translation via Cross-modal Progressive Training
• Learning Shared Semantic Space for Speech-to-Text Translation
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates