5 papers
FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models
Zhida Jiang, Zhaolong Xing, Yang Pei +14
Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design t…
The Hidden Power of Scaling Factor in LoRA Optimization
Zicheng Zhang, Haoran Li, Jiaxing Wang +10
In Low-Rank Adaptation (LoRA), the scaling factor is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this p…
Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
Hu Xu, Zhaolong Xing, Congcong Liu +5
Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects. We show that this conclus…
NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining
Zhida Jiang, Zhaolong Xing, Huichao Chai +12
Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to dat…
Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning
Zhida Jiang, Zhaolong Xing, Jiawei Lu +13
Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying networked infrastructure for large-scale MARL training remains underexplored. Exis…