28 citations · 83 across the 23 of their papers we have counts for
6 papers · 1 filter
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4
Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is p…
Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan +17
Training a family of large language models (LLMs), either from scratch or via iterative compression, is prohibitively expensive and inefficient, requiring separate training runs fo…
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aakshita Chandiramani +544
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemo…
Understanding the Effect of the Long Tail on Neural Network Compression
Harvey Dam, Vinu Joseph, Aditya Bhaskara +3
Network compression is now a mature sub-field of neural network research: over the last decade, significant progress has been made towards reducing the size of models and speeding…
Efficient Sparsely Activated Transformers
Salar Latifi, Saurav Muralidharan, Michael Garland
Transformer-based neural networks have achieved state-of-the-art task performance in a number of machine learning domains including natural language processing and computer vision.…
A Programmable Approach to Neural Network Compression
Vinu Joseph, Saurav Muralidharan, Animesh Garg +2
Deep neural networks (DNNs) frequently contain far more weights, represented at a higher precision, than are required for the specific task which they are trained to perform. Conse…