9 papers · 1 filter
Accelerating MoE Training through Expert Placement and Replica Coordination
Zhiqiang Tan, Maoxin Wang, Wenxiang Lin +4
Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token r…
LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL
Sijie Wang, Zhiqiang Tan, Xinrui Yang +1
Diffusion reinforcement learning (RL) has recently achieved significant success in post-training image and video generative models. However, most diffusion RL methods, including Da…
Laplacian Frequency Hierarchies for Efficient 3D Gaussian Splatting Training
Yixiong Yang, Sisheng Zhang, Qingsong Yan +2
A key bottleneck in 3D Gaussian Splatting training is the continual growth of Gaussian primitives, which increases optimization cost and slows convergence, especially at high resol…
Compass: Dissecting Communication and Computation Operators for Efficient LLM Training
Guangyu Xiang, Lin Zhang, Haoxuan Yu +3
Overlapping communication and computation operators is a common practice to hide communication overheads, accelerating large language models (LLMs) training on GPU clusters. Existi…
Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training
Zhiqiang Tan, Maoxin Wang, Sijie Wang +4
It is well established that the reasoning capabilities of large language models (LLMs) can be improved by applying reinforcement learning (RL) in a post-training stage. In a standa…
Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training
Guangyu Xiang, Xueze Kang, Minwei Zhao +4
High-quality video generation requires training Diffusion Transformers (DiTs) jointly on image and video data, posing a mixed-length sequence training problem across GPUs. Existing…
Xema: Efficient Diffusion Serving through Fine-Grained Memory Management and Auto-Configuration
Xueze Kang, Guangyu Xiang, Suyi Li +4
Diffusion models are increasingly deployed as production visual-generation services, where serving high-resolution image and long video generation is often limited by GPU memory. P…
KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding
Guangyu Xiang, Xueze Kang, Lin Zhang +4
LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deploy…
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
Wenxiang Lin, Juntao Huang, Luhan Zhang +5
Quantization is a key method for reducing the GPU memory requirement of training large language models (LLMs). Yet, current approaches are ineffective for 4-bit activations and 8-b…