3 citations · 9 across the 11 of their papers we have counts for
4 papers · 1 filter
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Shrey Pandit, Xuan-Phi Nguyen, Yiran Zhao +1
Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every pea…
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li +4
On-policy self-distillation aims to improve upon reinforcement learning from verifiable rewards (RLVR) by providing token-level scores derived from privileged information, such as…
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3
This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel paral…
Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
Xuan-Phi Nguyen, Shrey Pandit, Austin Xu +2
Mixture-of-Experts (MoE) models are typically pre-trained with explicit load-balancing constraints to ensure statistically balanced expert routing. Despite this, we observe that ev…