High Throughput Training of Deep Surrogates from Large Ensemble Runs

Abstract

Recent years have seen a surge in deep learning approaches to accelerate numerical solvers, which provide faithful but computationally intensive simulations of the physical world. These deep surrogates are generally trained in a supervised manner from limited amounts of data slowly generated by the same solver they intend to accelerate. We propose an open-source framework that enables the online training of these models from a large ensemble run of simulations. It leverages multiple levels of parallelism to generate rich datasets. The framework avoids I/O bottlenecks and storage issues by directly streaming the generated data. A training reservoir mitigates the inherent bias of streaming while maximizing GPU throughput. Experiment on training a fully connected network as a surrogate for the heat equation shows the proposed approach enables training on 8TB of data in 2 hours with an accuracy improved by 47% and a batch throughput multiplied by 13 compared to a traditional offline procedure.

arXiv commentThe International Conference for High Performance Computing, Networking, Storage, and Analysis, Nov 2023, Denver, CO, United States

References in corpus (27)

• Overcoming catastrophic forgetting in neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DGM: A deep learning algorithm for solving partial differential equations
• Fourier Neural Operator for Parametric Partial Differential Equations
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Dota 2 with Large Scale Deep Reinforcement Learning
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Distributed Prioritized Experience Replay
• A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems
• Massively Parallel Methods for Deep Reinforcement Learning
• RLlib: Abstractions for Distributed Reinforcement Learning
• Machine learning based spatio-temporal super resolution reconstruction of turbulent flows
• Parsl: Pervasive Parallel Programming in Python
• Probabilistic neural networks for fluid flow surrogate modeling and data recovery
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• AtlFast3: the next generation of fast simulation in ATLAS
• Revisiting Fundamentals of Experience Replay
• Simulation Intelligence: Towards a New Generation of Scientific Methods
• Message Passing Neural PDE Solvers
• PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
• Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing
• Transformer for Partial Differential Equations' Operator Learning
• Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale
• RADICAL-Cybertools: Middleware Building Blocks for Scalable Science
• Adaptive Ensemble Biomolecular Simulations at Scale
• EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
• Collect & Infer -- a fresh look at data-efficient Reinforcement Learning