RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning

Yibo Shen, Xudong Han, Xiaowei Zhu +2

Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-para…