26 citations · 47 across the 10 of their papers we have counts for
12 papers
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15
RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existi…
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aakshita Chandiramani +544
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemo…
LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
Venmugil Elango, Nidhi Bhatia, Roger Waleffe +13
Mixture of Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains…
NVIDIA Nemotron 3: Efficient and Open Intelligence
NVIDIA, :, Aaron Blakeman +356
We introduce the Nemotron 3 family of models - Nano, Super, and Ultra. These models deliver strong agentic, reasoning, and conversational capabilities. The Nemotron 3 family uses a…
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aaron Blakeman +311
We present Nemotron 3 Nano 30B-A3B, a Mixture-of-Experts hybrid Mamba-Transformer language model. Nemotron 3 Nano was pretrained on 25 trillion text tokens, including more than 3 t…
ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism
Venmugil Elango
Transformer-based models have emerged as a leading architecture for natural language processing, natural language generation, and image generation tasks. A fundamental element of t…
PaSE: Parallelization Strategies for Efficient DNN Training
Venmugil Elango
Training a deep neural network (DNN) requires substantial computational and memory requirements. It is common to use multiple devices to train a DNN to reduce the overall training…
Microscaling Data Formats for Deep Learning
Bita Darvish Rouhani, Ritchie Zhao, Ankit More +30
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that…
With Shared Microexponents, A Little Shifting Goes a Long Way
Bita Rouhani, Ritchie Zhao, Venmugil Elango +19
This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables compariso…
On Characterizing the Data Access Complexity of Programs
Venmugil Elango, Fabrice Rastello, Louis-Noel Pouchet +2
Technology trends will cause data movement to account for the majority of energy expenditure and execution time on emerging computers. Therefore, computational complexity will no l…
On Characterizing the Data Movement Complexity of Computational DAGs for Parallel Execution
Venmugil Elango, Fabrice Rastello, Louis-Noël Pouchet +2
Technology trends are making the cost of data movement increasingly dominant, both in terms of energy and time, over the cost of performing arithmetic operations in computer system…
Beyond Reuse Distance Analysis: Dynamic Analysis for Characterization of Data Locality Potential
Naznin Fauzia, Venmugil Elango, Mahesh Ravishankar +5
Emerging computer architectures will feature drastically decreased flops/byte (ratio of peak processing rate to memory bandwidth) as highlighted by recent studies on Exascale archi…