11 citations · 11 across the 3 of their papers we have counts for
10 papers
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
Sijie Wang, Zhengyu Qing, Zhiqiang Tan +6
Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models…
LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
Longteng Zhang, Lin Zhang, Shaohuai Shi +2
Fine-tuning large language models (LLMs) is crucial for improving their performance on downstream tasks, but full-parameter fine-tuning (Full-FT) is computationally expensive and m…
ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Wenxiang Lin, Xinglin Pan, Ruibo Fan +2
Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication…
ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
Xin He, Shunkang Zhang, Kaijie Tang +8
Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many…
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
Longteng Zhang, Sen Wu, Shuai Hou +7
Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in…
PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling
Sijie Wang, Qiang Wang, Shaohuai Shi
Video generation has been advancing rapidly, and diffusion transformer (DiT) based models have demonstrated remark- able capabilities. However, their practical deployment is of- te…