4 papers
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suf…
Dynamic Pyramid Network for Efficient Multimodal Large Language Model
Hao Ai, Kunyi Wang, Zezhou Wang +7
Multimodal large language models (MLLMs) have demonstrated impressive performance in various vision-language (VL) tasks, but their expensive computations still limit the real-world…
ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
Liangyu Zhao, Saeed Maleki, Yuanhong Wang +4
As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient comm…
OpenNet: Incremental Learning for Autonomous Driving Object Detection with Balanced Loss
Zezhou Wang, Guitao Cao, Xidong Xi +1
Automated driving object detection has always been a challenging task in computer vision due to environmental uncertainties. These uncertainties include significant differences in…