897 citations · 1.5k across the 14 of their papers we have counts for
14 papers
PartIR: Composing SPMD Partitioning Strategies for Machine Learning
Sami Alabed, Daniel Belov, Bart Chrzaszcz +14
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding. When strategies increase in comp…
The Foil: Capture-Avoiding Substitution With No Sharp Edges
Dougal Maclaurin, Alexey Radul, Adam Paszke
Correctly manipulating program terms in a compiler is surprisingly difficult because of the need to avoid name capture. The rapier from "Secrets of the Glasgow Haskell Compiler inl…
You Only Linearize Once: Tangents Transpose to Gradients
Alexey Radul, Adam Paszke, Roy Frostig +2
Automatic differentiation (AD) is conventionally understood as a family of distinct algorithms, rooted in two "modes" -- forward and reverse -- which are typically presented (and i…
Parallel Algebraic Effect Handlers
Ningning Xie, Daniel D. Johnson, Dougal Maclaurin +1
Algebraic effects and handlers support composable and structured control-flow abstraction. However, existing designs of algebraic effects often require effects to be executed seque…
Decomposing reverse-mode automatic differentiation
Roy Frostig, Matthew J. Johnson, Dougal Maclaurin +2
We decompose reverse-mode automatic differentiation into (forward-mode) linearization followed by transposition. Doing so isolates the essential difference between forward- and rev…
Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming
Adam Paszke, Daniel Johnson, David Duvenaud +5
We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level n…