961 citations · 1.4k across the 33 of their papers we have counts for
34 papers
Towards an AI Software Factory for Data Systems
Anna Pavlenko, Bogdan Crivat, Brandon Haynes +23
AI-assisted coding tools deliver significant acceleration of coding, but only limited impact across the end-to-end software development lifecycle (SDLC)--an Amdahl's law effect! In…
Mirage: Towards Low-interruption Services on Batch GPU Clusters with Reinforcement Learning
Qiyang Ding, Pengfei Zheng, Shreyas Kudari +2
Accommodating long-running deep learning (DL) training and inference jobs is challenging on GPU clusters that use traditional batch schedulers, such as Slurm. Given fixed wall cloc…
From FASTER to F2: Evolving Concurrent Key-Value Store Designs for Large Skewed Workloads
Konstantinos Kanellis, Badrish Chandramouli, Ted Hart +1
Modern large-scale services such as search engines, messaging platforms, and serverless functions, rely on key-value (KV) stores to maintain high performance at scale. When such se…
Does compressing activations help model parallel training?
Song Bian, Dacheng Li, Hongyi Wang +2
Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensiv…
Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning
Pengfei Zheng, Rui Pan, Tarannum Khan +2
Dynamic adaptation has become an essential technique in accelerating distributed machine learning (ML) training. Recent studies have shown that dynamically adjusting model structur…
Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems
Prasoon Sinha, Akhil Guliani, Rutwik Jain +3
Scientists are increasingly exploring and utilizing the massive parallelism of general-purpose accelerators such as GPUs for scientific breakthroughs. As a result, datacenters, hyp…
LlamaTune: Sample-Efficient DBMS Configuration Tuning
Konstantinos Kanellis, Cong Ding, Brian Kroth +3
Tuning a database system to achieve optimal performance on a given workload is a long-standing problem in the database community. A number of recent works have leveraged ML-based a…
BagPipe: Accelerating Deep Recommendation Model Training
Saurabh Agarwal, Chengpo Yan, Ziyi Zhang +1
Deep learning based recommendation models (DLRM) are widely used in several business critical applications. Training such recommendation models efficiently is challenging because t…
MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks
Roger Waleffe, Jason Mohoney, Theodoros Rekatsinas +1
We study training of Graph Neural Networks (GNNs) for large-scale graphs. We revisit the premise of using distributed training for billion-scale graphs and show that for graphs tha…
Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
Adarsh Kumar, Kausik Subramanian, Shivaram Venkataraman +1
Many organizations employ compute clusters equipped with accelerators such as GPUs and TPUs for training deep learning models in a distributed fashion. Training is resource-intensi…
KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
J. Gregory Pauloski, Qi Huang, Lei Huang +4
Kronecker-factored Approximate Curvature (K-FAC) has recently been shown to converge faster in deep neural network (DNN) training than stochastic gradient descent (SGD); however, K…
AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
Yuhan Liu, Saurabh Agarwal, Shivaram Venkataraman
With the rapid adoption of machine learning (ML), a number of domains now use the approach of fine tuning models which were pre-trained on a large corpus of data. However, our expe…
On the Utility of Gradient Compression in Distributed Training Systems
Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman +1
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent wo…
Accelerating Deep Learning Inference via Learned Caches
Arjun Balasubramanian, Adarsh Kumar, Yuhan Liu +3
Deep Neural Networks (DNNs) are witnessing increased adoption in multiple domains owing to their high accuracy in solving real-world problems. However, this high accuracy has been…
Marius: Learning Massive Graph Embeddings on a Single Machine
Jason Mohoney, Roger Waleffe, Yiheng Xu +2
We propose a new framework for computing the embeddings of large-scale graphs on a single machine. A graph embedding is a fixed length vector representation for each node (and/or e…
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
Saurabh Agarwal, Hongyi Wang, Kangwook Lee +2
Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use…
Move Fast and Meet Deadlines: Fine-grained Real-time Stream Processing with Cameo
Le Xu, Shivaram Venkataraman, Indranil Gupta +2
Resource provisioning in multi-tenant stream processing systems faces the dual challenges of keeping resource utilization high (without over-provisioning), and ensuring performance…
Accelerating Deep Learning Inference via Freezing
Adarsh Kumar, Arjun Balasubramanian, Shivaram Venkataraman +1
Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on real-world tasks. However, this increase in accuracy comes at the cost o…
Archipelago: A Scalable Low-Latency Serverless Platform
Arjun Singhvi, Kevin Houck, Arjun Balasubramanian +3
The increased use of micro-services to build web applications has spurred the rapid growth of Function-as-a-Service (FaaS) or serverless computing platforms. While FaaS simplifies…
Blink: Fast and Generic Collectives for Distributed ML
Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee +3
Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage…
Themis: Fair and Efficient GPU Cluster Scheduling
Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi +4
Modern distributed machine learning (ML) training workloads benefit significantly from leveraging GPUs. However, significant contention ensues when multiple such workloads are run…
Parity Models: A General Framework for Coding-Based Resilience in ML Inference
Jack Kosaian, K. V. Rashmi, Shivaram Venkataraman
Machine learning models are becoming the primary workhorses for many applications. Production services deploy models through prediction serving systems that take in queries and ret…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee +3
With widespread advances in machine learning, a number of large enterprises are beginning to incorporate machine learning models across a number of products. These models are typic…