Exascale Deep Learning for Scientific Inverse Problems

Abstract

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. We demonstrate our gradient reduction techniques in the context of training a Fully Convolutional Neural Network to approximate the solution of a longstanding scientific inverse problem in materials imaging. The efficient distributed training on a dataset size of 0.5 PB, produces a model capable of an atomically-accurate reconstruction of materials, and in the process reaching a peak performance of 2.15(4) EFLOPS${}_{16}$.

13 pages, 9 figures. Under review by the Systems and Machine Learning (SysML) Conference (SysML '20)

References in corpus (8)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Phase recovery and holographic image reconstruction using deep learning in neural networks
• cuDNN: Efficient Primitives for Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• Generating Long Sequences with Sparse Transformers
• Beyond Data and Model Parallelism for Deep Neural Networks
• Reconstruction of 3-D Atomic Distortions from Electron Microscopy with Deep Learning

Cited by in corpus (11)

• Review: Deep Learning in Electron Microscopy
• Is quantum computing green? An estimate for an energy-efficiency quantum advantage
• Reconstruction of nanoscale particles from single-shot wide-angle FEL diffractions patterns with physics-informed neural networks
• Deep learning Local Reduced Density Matrices for Many-body Hamiltonian Estimation
• Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study
• The Future of the Correlated Electron Problem
• Distributed Training and Optimization Of Neural Networks
• Towards the Development of Entropy-Based Anomaly Detection in an Astrophysics Simulation
• Integrating Deep Learning in Domain Sciences at Exascale
• Phase retrieval from 4-dimensional electron diffraction datasets
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research