39 citations · 98 across the 8 of their papers we have counts for
8 papers
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
Yuliang Liu, Shenggui Li, Jiarui Fang +3
In recent years, large-scale models have demonstrated state-of-the-art performance across various domains. However, training such models requires various techniques to address the…
Elixir: Train a Large Language Model on a Small GPU Cluster
Haichen Huang, Jiarui Fang, Hongxin Liu +2
In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requ…
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Jiangsu Du, Ziming Liu, Jiarui Fang +4
Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the tri…
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Jiarui Fang, Geng Zhang, Jiatong Han +5
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies. The embedding tables of DLRMs are too large to fit on GPU memory entirely. We propose a…
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
Jiarui Fang, Zilin Zhu, Shenggui Li +4
The pre-trained model (PTM) is revolutionizing Artificial Intelligence (AI) technology. However, the hardware requirement of PTM training is prohibitively high, making it a game fo…
TurboTransformers: An Efficient GPU Serving System For Transformer Models
Jiarui Fang, Yang Yu, Chengduo Zhao +1
The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, Transform…
swCaffe: a Parallel Framework for Accelerating Deep Learning Applications on Sunway TaihuLight
Jiarui Fang, Liandeng Li, Haohuan Fu +5
This paper reports our efforts on swCaffe, a highly efficient parallel framework for accelerating deep neural networks (DNNs) training on Sunway TaihuLight, the current fastest sup…
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Jiarui Fang, Haohuan Fu, Guangwen Yang +1
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since synchronizing a large number of gradients of the local model…