91 citations · 255 across the 16 of their papers we have counts for
16 papers
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
Seamless Communication, Loïc Barrault, Yu-An Chung +65
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed…
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks
Yun Tang, Anna Y. Sun, Hirofumi Inaguma +5
Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits…
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention
Xutai Ma, Hongyu Gong, Danni Liu +6
We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations.…
From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation
Danni Liu, Changhan Wang, Hongyu Gong +3
Speech-to-speech translation (S2ST) converts input speech to speech in another language. A challenge of delivering S2ST in real time is the accumulated delay between the translatio…
Direct speech-to-speech translation with discrete units
Ann Lee, Peng-Jen Chen, Changhan Wang +9
We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation…
SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation
Xutai Ma, Juan Pino, Philipp Koehn
Simultaneous text translation and end-to-end speech translation have recently made great progress but little work has combined these tasks together. We investigate how to adapt sim…
Streaming Simultaneous Speech Translation with Augmented Memory Transformer
Xutai Ma, Yongqiang Wang, Mohammad Javad Dousti +2
Transformer-based models have achieved state-of-the-art performance on speech translation tasks. However, the model architecture is not efficient enough for streaming scenarios sin…
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
Yun Tang, Juan Pino, Changhan Wang +2
Attention-based sequence-to-sequence modeling provides a powerful and elegant solution for applications that need to map one sequence to a different sequence. Its success heavily r…
fairseq S2T: Fast Speech-to-Text Modeling with fairseq
Changhan Wang, Yun Tang, Xutai Ma +4
We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's car…
SimulEval: An Evaluation Toolkit for Simultaneous Translation
Xutai Ma, Mohammad Javad Dousti, Changhan Wang +2
Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario where the model starts translating before reading the complete source input. Evalua…
Self-Training for End-to-End Speech Translation
Juan Pino, Qiantong Xu, Xutai Ma +2
One of the main challenges for end-to-end speech translation is data scarcity. We leverage pseudo-labels generated from unlabeled audio by a cascade and an end-to-end speech transl…
Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade
Juan Pino, Liezl Puzon, Jiatao Gu +3
For automatic speech translation (AST), end-to-end approaches are outperformed by cascaded models that transcribe with automatic speech recognition (ASR), then translate with machi…
Monotonic Multihead Attention
Xutai Ma, Juan Pino, James Cross +2
Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixe…
Broad-Coverage Semantic Parsing as Transduction
Sheng Zhang, Xutai Ma, Kevin Duh +1
We unify different broad-coverage semantic parsing tasks under a transduction paradigm, and propose an attention-based neural framework that incrementally builds a meaning represen…
AMR Parsing as Sequence-to-Graph Transduction
Sheng Zhang, Xutai Ma, Kevin Duh +1
We propose an attention-based model that treats AMR parsing as sequence-to-graph transduction. Unlike most AMR parsers that rely on pre-trained aligners, external semantic resource…
Optical analogy to quantum computation based on classical fields modulated pseudorandom phase sequences
Jian Fu, Xutai Ma, Wenjiang Li +1
We demonstrate that a tensor product structure and optical analogy of quantum entanglement can be obtained by introducing pseudorandom phase sequences into classical fields with tw…