29 citations · 60 across the 11 of their papers we have counts for
13 papers
TOAST: Fast and scalable auto-partitioning based on principled static analysis
Sami Alabed, Dominik Grewe, Norman Alexander Rink +5
Partitioning large machine learning models across distributed accelerator systems is a complex process, requiring a series of interdependent decisions that are further complicated…
PartIR: Composing SPMD Partitioning Strategies for Machine Learning
Sami Alabed, Daniel Belov, Bart Chrzaszcz +14
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding. When strategies increase in comp…
Efficient and Sound Differentiable Programming in a Functional Array-Processing Language
Amir Shaikhha, Mathieu Huot, Shabnam Ghasemirad +3
Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. This technique is considered as the de-facto standard for computi…
Automatic Discovery of Composite SPMD Partitioning Strategies in PartIR
Sami Alabed, Dominik Grewe, Juliana Franco +6
Large neural network models are commonly trained through a combination of advanced parallelism strategies in a single program, multiple data (SPMD) paradigm. For example, training…
Automap: Towards Ergonomic Automated Parallelism for ML Models
Michael Schaarschmidt, Dominik Grewe, Dimitrios Vytiniotis +8
The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline…
Memory-efficient array redistribution through portable collective communication
Norman A. Rink, Adam Paszke, Dimitrios Vytiniotis +1
Modern large-scale deep learning workloads highlight the need for parallel execution across many devices in order to fit model data into hardware accelerator memories. In these set…
Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning
Ningning Xie, Tamara Norman, Dominik Grewe +1
We present a novel characterization of the mapping of multiple parallelism forms (e.g. data and model parallelism) onto hierarchical accelerator systems that is hierarchy-aware and…
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
Mateusz Malinowski, Dimitrios Vytiniotis, Grzegorz Swirszcz +2
How can neural networks be trained on large-volume temporal data efficiently? To compute the gradients required to update parameters, backpropagation blocks computations until the…
Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming
Adam Paszke, Daniel Johnson, David Duvenaud +5
We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level n…
Efficient Differentiable Programming in a Functional Array-Processing Language
Amir Shaikhha, Andrew Fitzgibbon, Dimitrios Vytiniotis +2
We present a system for the automatic differentiation of a higher-order functional array-processing language. The core functional language underlying this system simultaneously sup…
AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
Alexander L. Gaunt, Matthew A. Johnson, Maik Riechert +4
New types of machine learning hardware in development and entering the market hold the promise of revolutionizing deep learning in a manner as profound as GPUs. However, existing s…
Measuring Neural Net Robustness with Constraints
Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos +3
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We pr…
Testing Noninterference, Quickly
Catalin Hritcu, Leonidas Lampropoulos, Antal Spector-Zabusky +5
Information-flow control mechanisms are difficult both to design and to prove correct. To reduce the time wasted on doomed proof attempts due to broken definitions, we advocate mod…