12 papers
Motif-Mamba: network motif improved mamba for long-range sequence modeling
Chonghe Hao, Yue Sun, Jian Zhang +4
Efficient long-sequence modeling remains a central challenge for large language models, as self-attention scales quadratically with sequence length. Mamba offers a linear-time alte…
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aaron Blakeman +571
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 t…
: A "Spot the Difference" Challenge for Large Multimodal Models
Kewei Wei, Bocheng Hu, Jie Cao +13
Modern Large Multimodal Models (LMMs) have demonstrated extraordinary ability in static image and single-state spatial-temporal understanding. However, their capacity to comprehend…
PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang +9
Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers…
LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Models
Tiesunlong Shen, Rui Mao, Jin Wang +4
Aligning Large Language Models (LLMs) with human preferences is critical, yet traditional fine-tuning methods are computationally expensive and inflexible. While test-time alignmen…
MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
Lu Zhao, Rong Shi, Shaoqing Zhang +21
The training of large-scale Mixture of Experts (MoE) models faces a critical memory bottleneck due to severe load imbalance caused by dynamic token routing. This imbalance leads to…