Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
arXiv:2102.05766
Abstract
Recently, representation learning for text and speech has successfully improved many language related tasks. However, all existing methods suffer from two limitations: (a) they only learn from one input modality, while a unified representation for both speech and text is needed by tasks such as end-to-end speech translation, and as a result,(b) they can not exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data.To address these problems, we propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly learns a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and even pure speech and text data. Within this cross-modal representation learning framework, we further present an end-to-end model for Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three translation directions show that by fine-tuning from FAT-MLM, our proposed speech translation models substantially improve translation quality by up to +5.9 BLEU.
References in corpus (10)
- Cross-lingual Language Model Pretraining
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- ERNIE: Enhanced Representation through Knowledge Integration
- TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
- Bridging the Modality Gap for Speech-to-Text Translation
- End-to-End Speech Translation with Knowledge Distillation
- SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
- MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation
- Consecutive Decoding for Speech-to-text Translation
- Neural Lattice-to-Sequence Models for Uncertain Inputs