22 citations · 27 across the 6 of their papers we have counts for
6 papers
ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Tenghui Ma, Jihu Guo, Wei Gao +4
Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual devices lead to performance skew across devic…
PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
Wei Gao, Peng Sun, Dmitrii Ustiugov +2
Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Se…
Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
Wei Gao, Xinyu Zhou, Peng Sun +2
Key-Value cache (\texttt{KV} \texttt{cache}) compression has emerged as a promising technique to optimize Large Language Model (LLM) serving. It primarily decreases the memory cons…
SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
Qiaoling Chen, Shenggui Li, Wei Gao +3
In recent years, Large Language Models (LLMs) have exhibited remarkable capabilities, driving advancements in real-world applications. However, training LLMs on increasingly long i…
Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision
Wei Gao, Qinghao Hu, Zhisheng Ye +5
Deep learning (DL) shows its prosperity in a wide variety of fields. The development of a DL model is a time-consuming and resource-intensive procedure. Hence, dedicated GPU accele…
Privacy-preserving Collaborative Learning with Automatic Transformation Search
Wei Gao, Shangwei Guo, Tianwei Zhang +3
Collaborative learning has gained great popularity due to its benefit of data privacy protection: participants can jointly train a Deep Learning model without sharing their trainin…