507 citations · 1k across the 22 of their papers we have counts for
23 papers
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Ziming Liu, Shenggan Cheng, Haotian Zhou +1
Large-scale language models have become increasingly challenging and expensive to train. Among various methods addressing this issue, Pipeline Parallelism has been widely employed…
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
Yuliang Liu, Shenggui Li, Jiarui Fang +3
In recent years, large-scale models have demonstrated state-of-the-art performance across various domains. However, training such models requires various techniques to address the…
ATP: Adaptive Tensor Parallelism for Foundation Models
Shenggan Cheng, Ziming Liu, Jiangsu Du +1
Foundation models have impressive performance and generalization capabilities across a wide range of applications. The increasing size of the models introduces great challenges for…
MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs
Huaizheng Zhang, Yuanming Li, Wencong Xiao +7
New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This tech…
Elixir: Train a Large Language Model on a Small GPU Cluster
Haichen Huang, Jiarui Fang, Hongxin Liu +2
In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requ…
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Jiangsu Du, Ziming Liu, Jiarui Fang +4
Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the tri…