51 citations · 209 across the 15 of their papers we have counts for
16 papers
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Zichang Liu, Jue Wang, Tri Dao +8
Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference t…
Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning
Yuxin Tang, Zhimin Ding, Dimitrije Jankov +3
The relational data model was designed to facilitate large-scale data management and analytics. We consider the problem of how to differentiate computations expressed relationally.…
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Ying Sheng, Lianmin Zheng, Binhang Yuan +11
The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand f…
Stochastic Gradient Descent without Full Data Shuffle
Lijie Xu, Shuang Qiu, Binhang Yuan +9
Stochastic gradient descent (SGD) is the cornerstone of modern machine learning (ML) systems. Despite its computational efficiency, SGD requires random data access that is inherent…
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Jue Wang, Binhang Yuan, Luka Rimanic +5
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive…
Decentralized Training of Foundation Models in Heterogeneous Environments
Binhang Yuan, Yongjun He, Jared Quincy Davis +6
Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months. These models are typicall…
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters
Xiangru Lian, Binhang Yuan, Xuefeng Zhu +24
Deep learning based models have dominated the current landscape of production recommender systems. Furthermore, recent years have witnessed an exponential growth of the model scale…
BAGUA: Scaling up Distributed Learning with System Relaxations
Shaoduo Gan, Xiangru Lian, Rui Wang +11
Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style…
Tensor Relational Algebra for Machine Learning System Design
Binhang Yuan, Dimitrije Jankov, Jia Zou +3
We consider the question: what is the abstraction that should be implemented by the computational engine of a machine learning system? Current machine learning systems typically pu…
Lachesis: Automatic Partitioning for UDF-Centric Analytics
Jia Zou, Amitabh Das, Pratik Barhate +4
Persistent partitioning is effective in avoiding expensive shuffling operations. However it remains a significant challenge to automate this process for Big Data analytics workload…
A Federated Learning Framework for Healthcare IoT devices
Binhang Yuan, Song Ge, Wenhui Xing
The Internet of Things (IoT) revolution has shown potential to give rise to many medical applications with access to large volumes of healthcare data collected by IoT devices. Howe…
Distributed Learning of Deep Neural Networks using Independent Subnet Training
Binhang Yuan, Cameron R. Wolfe, Chen Dun +3
Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning parti…
Diagnosing Cardiac Abnormalities from 12-Lead Electrocardiograms Using Enhanced Deep Convolutional Neural Networks
Binhang Yuan, Wenhui Xing
We train an enhanced deep convolutional neural network in order to identify eight cardiac abnormalities from the standard 12-lead electrocardiograms (ECGs) using the dataset of 140…
Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
Dimitrije Jankov, Shangyu Luo, Binhang Yuan +4
A number of popular systems, most notably Google's TensorFlow, have been implemented from the ground up to support machine learning tasks. We consider how to make a very small set…
WaveletAE: A Wavelet-enhanced Autoencoder for Wind Turbine Blade Icing Detection
Binhang Yuan, Chen Wang, Chen Luo +4
Wind power, as an alternative to burning fossil fuels, is abundant and inexhaustible. To fully utilize wind power, wind farms are usually located in areas of high altitude and faci…
PlinyCompute: A Platform for High-Performance, Distributed, Data-Intensive Tool Development
Jia Zou, R. Matthew Barnett, Tania Lorido-Botran +6
This paper describes PlinyCompute, a system for development of high-performance, data-intensive, distributed computing tools and libraries. In the large, PlinyCompute presents the…