fairseq S2T: Fast Speech-to-Text Modeling with fairseq

Abstract

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Post-conference updates (accepted to AACL 2020 Demo)

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• Monotonic Multihead Attention
• Thinking Slow about Latency Evaluation for Simultaneous Machine Translation

Cited by in corpus (21)

• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR
• Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
• End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• Does Simultaneous Speech Translation need Simultaneous Models?
• Attention as a Guide for Simultaneous Speech Translation
• NeurST: Neural Speech Translation Toolkit
• Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
• Dealing with training and test segmentation mismatch: FBK@IWSLT2021
• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
• Simultaneous Speech Translation for Live Subtitling: from Delay to Display
• Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
• Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates
• Multilingual Speech Translation with Unified Transformer: Huawei Noah's Ark Lab at IWSLT 2021
• RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
• Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation