163 citations · 461 across the 19 of their papers we have counts for
3 papers · 1 filter
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training
Hiroki Naganuma, Xinzhi Zhang, Man-Chung Yue +4
Following AI scaling trends, frontier models continue to grow in size and continue to be trained on larger datasets. Training these models requires huge investments in exascale com…
Model-Parallel Fourier Neural Operators as Learned Surrogates for Large-Scale Parametric PDEs
Thomas J. Grady, Rishi Khan, Mathias Louboutin +5
Fourier neural operators (FNOs) are a recently introduced neural network architecture for learning solution operators of partial differential equations (PDEs), which have been show…
A Linear Algebraic Approach to Model Parallelism in Deep Learning
Russell J. Hewett, Thomas J. Grady
Training deep neural networks (DNNs) in large-cluster computing environments is increasingly necessary, as networks grow in size and complexity. Local memory and processing limitat…