961 citations · 1.4k across the 33 of their papers we have counts for
13 papers · 1 filter
Does compressing activations help model parallel training?
Song Bian, Dacheng Li, Hongyi Wang +2
Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensiv…
MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks
Roger Waleffe, Jason Mohoney, Theodoros Rekatsinas +1
We study training of Graph Neural Networks (GNNs) for large-scale graphs. We revisit the premise of using distributed training for billion-scale graphs and show that for graphs tha…
KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
J. Gregory Pauloski, Qi Huang, Lei Huang +4
Kronecker-factored Approximate Curvature (K-FAC) has recently been shown to converge faster in deep neural network (DNN) training than stochastic gradient descent (SGD); however, K…
AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
Yuhan Liu, Saurabh Agarwal, Shivaram Venkataraman
With the rapid adoption of machine learning (ML), a number of domains now use the approach of fine tuning models which were pre-trained on a large corpus of data. However, our expe…
Accelerating Deep Learning Inference via Learned Caches
Arjun Balasubramanian, Adarsh Kumar, Yuhan Liu +3
Deep Neural Networks (DNNs) are witnessing increased adoption in multiple domains owing to their high accuracy in solving real-world problems. However, this high accuracy has been…
Marius: Learning Massive Graph Embeddings on a Single Machine
Jason Mohoney, Roger Waleffe, Yiheng Xu +2
We propose a new framework for computing the embeddings of large-scale graphs on a single machine. A graph embedding is a fixed length vector representation for each node (and/or e…
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
Saurabh Agarwal, Hongyi Wang, Kangwook Lee +2
Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use…
Accelerating Deep Learning Inference via Freezing
Adarsh Kumar, Arjun Balasubramanian, Shivaram Venkataraman +1
Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on real-world tasks. However, this increase in accuracy comes at the cost o…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Learning a Code: Machine Learning for Approximate Non-Linear Coded Computation
Jack Kosaian, K. V. Rashmi, Shivaram Venkataraman
Machine learning algorithms are typically run on large scale, distributed compute infrastructure that routinely face a number of unavailabilities such as failures and temporary slo…
KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics
Evan R. Sparks, Shivaram Venkataraman, Tomer Kaftan +2
Modern advanced analytics applications make use of machine learning techniques and contain multiple steps of domain-specific and general-purpose processing with high resource requi…
Large Scale Kernel Learning using Block Coordinate Descent
Stephen Tu, Rebecca Roelofs, Shivaram Venkataraman +1
We demonstrate that distributed block coordinate descent can quickly solve kernel regression and classification problems with millions of data points. Armed with this capability, w…
MLlib: Machine Learning in Apache Spark
Xiangrui Meng, Joseph Bradley, Burak Yavuz +13
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's ope…