261 citations · 682 across the 43 of their papers we have counts for
44 papers
SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separation
Helin Wang, Bowen Shi, Andros Tjandra +6
The performance evaluation remains a complex challenge in audio separation, and existing evaluation metrics are often misaligned with human perception, course-grained, relying on g…
SAM Audio: Segment Anything in Audio
Bowen Shi, Andros Tjandra, John Hoffman +11
General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separ…
MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
Alon Ziv, Sanyuan Chen, Andros Tjandra +3
A key challenge in music generation models is their lack of direct alignment with human preferences, as music evaluation is inherently subjective and varies widely across individua…
The AudioMOS Challenge 2025
Wen-Chin Huang, Hui Wang, Cheng Liu +6
This is the summary paper for the AudioMOS Challenge 2025, the very first challenge for automatic subjective quality prediction for synthetic audio. The challenge consists of three…
Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
Andros Tjandra, Yi-Chiao Wu, Baishan Guo +10
The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural…
Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
Mu Yang, Bowen Shi, Matthew Le +2
This work focuses on improving Text-To-Audio (TTA) generation on zero-shot and few-shot settings (i.e. generating unseen or uncommon audio events). Inspired by the success of Retri…
MusicFlow: Cascaded Flow Matching for Text Guided Music Generation
K R Prajwal, Bowen Shi, Matthew Lee +8
We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music aud…
Movie Gen: A Cast of Media Foundation Models
Adam Polyak, Amit Zohar, Andrew Brown +85
We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabili…
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning
Chung-Ming Chien, Andros Tjandra, Apoorv Vyas +3
As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adap…
IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages
Tahir Javed, Janki Atul Nawale, Eldho Ittan George +18
We present INDICVOICES, a dataset of natural and spontaneous speech containing a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio from 16237 speaker…
Audiobox: Unified Audio Generation with Natural Language Prompts
Apoorv Vyas, Bowen Shi, Matthew Le +21
Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing t…
Generative Pre-training for Speech with Flow Matching
Alexander H. Liu, Matt Le, Apoorv Vyas +3
Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high…
Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model
Jiamin Xie, Ke Li, Jinxi Guo +7
Neural network pruning offers an effective method for compressing a multilingual automatic speech recognition (ASR) model with minimal performance loss. However, it entails several…
Scaling Speech Technology to 1,000+ Languages
Vineel Pratap, Andros Tjandra, Bowen Shi +13
Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to ab…
SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain
Heli Qi, Sashi Novitasari, Andros Tjandra +2
This paper introduces SpeeChain, an open-source Pytorch-based toolkit designed to develop the machine speech chain for large-scale use. This first release focuses on the TTS-to-ASR…
Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities
Andros Tjandra, Nayan Singhal, David Zhang +4
End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high…
Voice-preserving Zero-shot Multiple Accent Conversion
Mumin Jin, Prashant Serai, Jilong Wu +3
Most people who have tried to learn a foreign language would have experienced difficulties understanding or speaking with a native speaker's accent. For native speakers, understand…
Learning ASR pathways: A sparse multilingual ASR model
Mu Yang, Andros Tjandra, Chunxi Liu +3
Neural network pruning compresses automatic speech recognition (ASR) models effectively. However, in multilingual ASR, language-agnostic pruning may lead to severe performance drop…
Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation
Rendi Chevi, Radityo Eko Prasojo, Alham Fikri Aji +2
Several solutions for lightweight TTS have shown promising results. Still, they either rely on a hand-crafted design that reaches non-optimum size or use a neural architecture sear…
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Arun Babu, Changhan Wang, Andros Tjandra +10
This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a mill…
Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings
Jialu Li, Vimal Manohar, Pooja Chitkara +5
Speech recognition models often obtain degraded performance when tested on speech with unseen accents. Domain-adversarial training (DAT) and multi-task learning (MTL) are two commo…
Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks
Sangeeta Srivastava, Yun Wang, Andros Tjandra +4
Representation learning from unlabeled data has been of major interest in artificial intelligence research. While self-supervised speech representation learning has been popular in…
Improved Language Identification Through Cross-Lingual Self-Supervised Learning
Andros Tjandra, Diptanu Gon Choudhury, Frank Zhang +6
Language identification greatly impacts the success of downstream tasks such as automatic speech recognition. Recently, self-supervised speech representations learned by wav2vec 2.…
Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis
Sashi Novitasari, Andros Tjandra, Sakriani Sakti +1
Even though over seven hundred ethnic languages are spoken in Indonesia, the available technology remains limited that could support communication within indigenous communities as…