131 citations · 138 across the 4 of their papers we have counts for
10 papers · 1 filter
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Jiangfei Duan, Shuo Zhang, Zerui Wang +13
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and sign…
InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Qiaoling Chen, Diandian Gu, Guoteng Wang +8
Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are n…
AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
Qiaoling Chen, Qinghao Hu, Guoteng Wang +8
Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer…
A Simulation Platform for Multi-tenant Machine Learning Services on Thousands of GPUs
Ruofan Liang, Bingsheng He, Shengen Yan +1
Multi-tenant machine learning services have become emerging data-intensive workloads in data centers with heavy usage of GPU resources. Due to the large scale, many tuning paramete…
Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters
Qinghao Hu, Peng Sun, Shengen Yan +2
Modern GPU datacenters are critical for delivering Deep Learning (DL) models and services in both the research community and industry. When operating a datacenter, optimization of…
ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
Yizheng Huang, Huaizheng Zhang, Yonggang Wen +2
MLOps is about taking experimental ML models to production, i.e., serving the models to actual users. Unfortunately, existing ML serving systems do not adequately handle the dynami…