116 citations · 226 across the 16 of their papers we have counts for
19 papers
Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero +81
Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent…
SSR: Alignment-Aware Modality Connector for Speech Language Models
Weiting Tan, Hirofumi Inaguma, Ning Dong +2
Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We…
Seamless: Multilingual Expressive and Streaming Speech Translation
Seamless Communication, Loïc Barrault, Yu-An Chung +62
Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this wo…
Efficient Monotonic Multihead Attention
Xutai Ma, Anna Sun, Siqi Ouyang +2
We introduce the Efficient Monotonic Multihead Attention (EMMA), a state-of-the-art simultaneous translation model with numerically-stable and unbiased monotonic alignment estimati…
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
Seamless Communication, Loïc Barrault, Yu-An Chung +65
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed…
Scaling Speech Technology to 1,000+ Languages
Vineel Pratap, Andros Tjandra, Bowen Shi +13
Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to ab…
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks
Yun Tang, Anna Y. Sun, Hirofumi Inaguma +5
Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits…
Efficient Speech Representation Learning with Low-Bit Quantization
Ching-Feng Yeh, Wei-Ning Hsu, Paden Tomasello +1
With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity. In effort to improve the efficiency…
Continual Learning for On-Device Speech Recognition using Disentangled Conformers
Anuj Diwan, Ching-Feng Yeh, Wei-Ning Hsu +4
Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encoun…
Speech-to-Speech Translation For A Real-world Unwritten Language
Peng-Jen Chen, Kevin Tran, Yilin Yang +13
We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard te…
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Paden Tomasello, Akshat Shrivastava, Daniel Lazar +12
End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. It promises to improve the performance of assistant systems by leveraging a…
Deliberation Model for On-Device Spoken Language Understanding
Duc Le, Akshat Shrivastava, Paden Tomasello +4
We propose a novel deliberation-based approach to end-to-end (E2E) spoken language understanding (SLU), where a streaming automatic speech recognition (ASR) model produces the firs…
Generative Spoken Dialogue Language Modeling
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet +8
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled wi…
textless-lib: a Library for Textless Spoken Language Processing
Eugene Kharitonov, Jade Copet, Kushal Lakhotia +8
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this p…
Flashlight: Enabling Innovation in Tools for Machine Learning
Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko +11
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become chall…
Self-training and Pre-training are Complementary for Speech Recognition
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko +5
Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data. However, it is not clear whether they l…
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap +5
Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single da…
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
Vineel Pratap, Anuroop Sriram, Paden Tomasello +4
We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all si…
DSCnet: Replicating Lidar Point Clouds with Deep Sensor Cloning
Paden Tomasello, Sammy Sidhu, Anting Shen +6
Convolutional neural networks (CNNs) have become increasingly popular for solving a variety of computer vision tasks, ranging from image classification to image segmentation. Recen…