11 papers
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding
Matan Rusanovsky, Yoav Miron, Roy Uziel +4
Speculative decoding accelerates language-model inference by drafting future tokens that the target model verifies in parallel. A diffusion-style block head such as DFlash is an at…
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67
We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under…
Cybersecurity is the True Frontier for Generative AI Success or Failure
Edward Raff, Maor Ashkenazi, Sagar Samtani +2
Cybersecurity is a real-life test-bed for many machine learning problems at once, especially when considering modern strides in using Large Language Models (LLMs) to automate proce…
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aaron Blakeman +571
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 t…
SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
Talor Abramovich, Maor Ashkenazi, Izzy Putterman +5
Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is in…
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15
RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existi…