47 citations · 74 across the 6 of their papers we have counts for
8 papers
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Ying Sheng, Lianmin Zheng, Binhang Yuan +11
The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand f…
Stochastic Gradient Descent without Full Data Shuffle
Lijie Xu, Shuang Qiu, Binhang Yuan +9
Stochastic gradient descent (SGD) is the cornerstone of modern machine learning (ML) systems. Despite its computational efficiency, SGD requires random data access that is inherent…
SHiFT: An Efficient, Flexible Search Engine for Transfer Learning
Cedric Renggli, Xiaozhe Yao, Luka Kolar +3
Transfer learning can be seen as a data- and compute-efficient alternative to training models from scratch. The emergence of rich model repositories, such as TensorFlow Hub, enable…
Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments
Leonel Aguilar, Michal Gath-Morad, Jascha Grübel +7
A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse te…
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters
Xiangru Lian, Binhang Yuan, Xuefeng Zhu +24
Deep learning based models have dominated the current landscape of production recommender systems. Furthermore, recent years have witnessed an exponential growth of the model scale…
BAGUA: Scaling up Distributed Learning with System Relaxations
Shaoduo Gan, Xiangru Lian, Rui Wang +11
Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style…
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan +6
Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, commu…
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari +4
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compat…