7 citations · 17 across the 4 of their papers we have counts for
5 papers
PartIR: Composing SPMD Partitioning Strategies for Machine Learning
Sami Alabed, Daniel Belov, Bart Chrzaszcz +14
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding. When strategies increase in comp…
Automatic Discovery of Composite SPMD Partitioning Strategies in PartIR
Sami Alabed, Dominik Grewe, Juliana Franco +6
Large neural network models are commonly trained through a combination of advanced parallelism strategies in a single program, multiple data (SPMD) paradigm. For example, training…
Automap: Towards Ergonomic Automated Parallelism for ML Models
Michael Schaarschmidt, Dominik Grewe, Dimitrios Vytiniotis +8
The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline…
Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning
Ningning Xie, Tamara Norman, Dominik Grewe +1
We present a novel characterization of the mapping of multiple parallelism forms (e.g. data and model parallelism) onto hierarchical accelerator systems that is hierarchy-aware and…
Acme: A Research Framework for Distributed Reinforcement Learning
Matthew W. Hoffman, Bobak Shahriari, John Aslanides +36
Deep reinforcement learning (RL) has led to many recent and groundbreaking advances. However, these advances have often come at the cost of both increased scale in the underlying a…