1.1k citations · 1.5k across the 16 of their papers we have counts for
17 papers
To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
Meghana Madhyastha, Daniel Haziza, Jesse Cai +3
Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Fo…
The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes
Redacted by arXiv
This document consolidates publicly reported technical details about Metas Llama 4 model family. It summarizes (i) released variants (Scout and Maverick) and the broader herd conte…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Set Block Decoding is a Language Model Inference Accelerator
Itai Gat, Heli Ben-Hamu, Marton Havasi +6
Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory co…
DINOv3
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer +23
Self-supervised learning holds the promise of eliminating the need for manual data annotation, enabling models to scale effortlessly to massive datasets and larger architectures. B…
Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Bangsheng Tang, Carl Chengyan Fu, Fei Kou +35
Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering c…
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
Daniel Haziza, Timothy Chou, Dhruv Choudhary +7
In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and infe…
Memory Layers at Scale
Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza +3
Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement comp…
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference
Yejin Lee, Anna Sun, Basil Hosmer +27
Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system…
IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages
Tahir Javed, Janki Atul Nawale, Eldho Ittan George +18
We present INDICVOICES, a dataset of natural and spontaneous speech containing a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio from 16237 speaker…
Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on Aerial Lidar
Jamie Tolan, Hung-I Yang, Ben Nosarzewski +13
Vegetation structure mapping is critical for understanding the global carbon cycle and monitoring nature-based approaches to climate adaptation and mitigation. Repeated measurement…
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni +23
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. Thes…
Efficient conditioned face animation using frontally-viewed embedding
Maxime Oquab, Daniel Haziza, Ludovic Schwartz +5
As the quality of few shot facial animation from landmarks increases, new applications become possible, such as ultra low bandwidth video chat compression with a high degree of rea…
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
Changhan Wang, Morgane Rivière, Ann Lee +6
We introduce VoxPopuli, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages. It is the largest open data to date for unsupervised repre…
Low Bandwidth Video-Chat Compression using Deep Generative Models
Maxime Oquab, Pierre Stock, Oran Gafni +9
To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver's de…
Population Based Training for Data Augmentation and Regularization in Speech Recognition
Daniel Haziza, Jérémy Rapin, Gabriel Synnaeve
Varying data augmentation policies and regularization over the course of optimization has led to performance improvements over using fixed values. We show that population based tra…
Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning
Francois Belletti, Daniel Haziza, Gabriel Gomes +1
This article shows how the recent breakthroughs in Reinforcement Learning (RL) that have enabled robots to learn to play arcade video games, walk or assemble colored bricks, can be…