9 papers
To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
Meghana Madhyastha, Daniel Haziza, Jesse Cai +3
Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Fo…
The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes
Redacted by arXiv
This document consolidates publicly reported technical details about Metas Llama 4 model family. It summarizes (i) released variants (Scout and Maverick) and the broader herd conte…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Set Block Decoding is a Language Model Inference Accelerator
Itai Gat, Heli Ben-Hamu, Marton Havasi +6
Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory co…
DINOv3
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer +23
Self-supervised learning holds the promise of eliminating the need for manual data annotation, enabling models to scale effortlessly to massive datasets and larger architectures. B…
Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Bangsheng Tang, Carl Chengyan Fu, Fei Kou +35
Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering c…