Publications (25)
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
Jared Fernandez, Jacob Kahn, Clara Na +2
Increased focus on the computational efficiency of NLP systems has motivated the design of efficient model architectures and improvements to underlying hardware accelerators. Howev…
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Jacob Kahn, Morgane Rivière, Weiyi Zheng +12
We introduce a new collection of spoken English audio suitable for training speech recognition systems under limited or no supervision. It is derived from open-source audio books f…
Scaling Up Online Speech Recognition Using ConvNets
Vineel Pratap, Qiantong Xu, Jacob Kahn +6
We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC). We improve the core TDS a…
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski +8
Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist l…
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Chunting Zhou, Lili Yu, Arun Babu +7
We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token predicti…
Flashlight: Enabling Innovation in Tools for Machine Learning
Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko +11
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become chall…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Xi Victoria Lin, Xilun Chen, Mingda Chen +9
Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing ap…
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn +6
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functio…
Code World Model Preparedness Report
Daniel Song, Peter Ney, Cristina Menghini +21
This report documents the preparedness assessment of Code World Model (CWM), a model for code generation and reasoning about code from Meta. We conducted pre-release testing across…
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
Jared Fernandez, Luca Wehrstedt, Leonid Shamis +5
Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To devel…
SlimIPL: Language-Model-Free Iterative Pseudo-Labeling
Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn +2
Recent results in end-to-end automatic speech recognition have demonstrated the efficacy of pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Clas…
Iterative Pseudo-Labeling for Speech Recognition
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn +3
Pseudo-labeling has recently shown promise in end-to-end automatic speech recognition (ASR). We study Iterative Pseudo-Labeling (IPL), a semi-supervised algorithm which efficiently…
Altogether: Image Captioning via Re-aligning Alt-text
Hu Xu, Po-Yao Huang, Xiaoqing Ellen Tan +10
This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ign…
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference
Yejin Lee, Anna Sun, Basil Hosmer +27
Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system…
PRISM: Evaluating POSIX Storage Systems for AI Research Workflows
Adithya Kumar, Aditya Basu, Jacob Kahn +3
The rapid advancement of AI research is driven by massive investments in GPU clusters, yet the critical role of storage systems in enabling efficient research workflows is often ov…
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks
Benoit Steiner, Mostafa Elhoushi, Jacob Kahn +1
The size of deep neural networks has grown exponentially in recent years. Unfortunately, hardware devices have not kept pace with the rapidly increasing memory requirements. To cop…
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma +8
We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowled…
Reasoning over Public and Private Data in Retrieval-Based Systems
Simran Arora, Patrick Lewis, Angela Fan +2
Users and organizations are generating ever-increasing amounts of private data from a wide range of sources. Incorporating private data is important to personalize open-domain appl…
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch
Jeff Hwang, Moto Hira, Caroline Chen +21
TorchAudio is an open-source audio and speech processing library built for PyTorch. It aims to accelerate the research and development of audio and speech technologies by providing…
Differentiable Weighted Finite-State Transducers
Awni Hannun, Vineel Pratap, Jacob Kahn +1
We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time. Through the separation…
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap +5
Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single da…
Agentic Operator Generation for ML ASICs
Alec M. Hammond, Aram Markosyan, Aman Dontula +15
We present TritorX, an agentic AI system designed to generate functionally correct Triton PyTorch ATen kernels at scale for emerging accelerator platforms. TritorX integrates open-…
Self-Training for End-to-End Speech Recognition
Jacob Kahn, Ann Lee, Awni Hannun
We revisit self-training in the context of end-to-end speech recognition. We demonstrate that training with pseudo-labels can substantially improve the accuracy of a baseline model…
wav2letter++: The Fastest Open-source Speech Recognition System
Vineel Pratap, Awni Hannun, Qiantong Xu +5
This paper introduces wav2letter++, the fastest open-source deep learning speech recognition framework. wav2letter++ is written entirely in C++, and uses the ArrayFire tensor libra…