29 citations · 85 across the 11 of their papers we have counts for
6 papers · 1 filter
DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
Chenyu Jiang, Zhenkun Cai, Ye Tian +3
Context parallelism has emerged as a key technique to support long-context training, a growing trend in generative AI for modern large models. However, existing context parallel me…
Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping
Chenyu Jiang, Ye Tian, Zhen Jia +3
The Mixture-of-Expert (MoE) technique plays a crucial role in expanding the size of DNN model parameters. However, it faces the challenge of extended all-to-all communication laten…
DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines
Chenyu Jiang, Zhen Jia, Shuai Zheng +2
Multi-task model training has been adopted to enable a single deep neural network model (often a large language model) to handle multiple tasks (e.g., question answering and text s…
Is Network the Bottleneck of Distributed Training?
Zhen Zhang, Chaokun Chang, Haibin Lin +3
Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand wheth…
Efficient Execution of Quantized Deep Learning Models: A Compiler Approach
Animesh Jain, Shoubhik Bhattacharya, Masahiro Masuda +2
A growing number of applications implement predictive functions using deep learning models, which require heavy use of compute and memory. One popular technique for increasing reso…
A Unified Optimization Approach for CNN Model Inference on Integrated GPUs
Leyuan Wang, Zhi Chen, Yizhi Liu +4
Modern deep learning applications urge to push the model inference taking place at the edge devices for multiple reasons such as achieving shorter latency, relieving the burden of…