Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
arXiv:2310.03724 · doi:10.21437/Interspeech.2023-2484
Abstract
Recent research has shown that independently trained encoders and decoders, combined through a shared fixed-size representation, can achieve competitive performance in speech-to-text translation. In this work, we show that this type of approach can be further improved with multilingual training. We observe significant improvements in zero-shot cross-modal speech translation, even outperforming a supervised approach based on XLSR for several languages.
References in corpus (7)
- Robust Speech Recognition via Large-Scale Weak Supervision
- Beyond English-Centric Multilingual Machine Translation
- The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
- mSLAM: Massively multilingual joint pre-training for speech and text
- DeepNet: Scaling Transformers to 1,000 Layers
- CoVoST 2 and Massively Multilingual Speech-to-Text Translation
- SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation