21 citations · 30 across the 4 of their papers we have counts for
4 papers
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Jiangsu Du, Ziming Liu, Jiarui Fang +4
Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the tri…
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Jiarui Fang, Geng Zhang, Jiatong Han +5
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies. The embedding tables of DLRMs are too large to fit on GPU memory entirely. We propose a…
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Shenggui Li, Hongxin Liu, Zhengda Bian +5
The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice…
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal +2
Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focus…