Differentiable modeling to unify machine learning and physical models and advance Geosciences

Abstract

Process-Based Modeling (PBM) and Machine Learning (ML) are often perceived as distinct paradigms in the geosciences. Here we present differentiable geoscientific modeling as a powerful pathway toward dissolving the perceived barrier between them and ushering in a paradigm shift. For decades, PBM offered benefits in interpretability and physical consistency but struggled to efficiently leverage large datasets. ML methods, especially deep networks, presented strong predictive skills yet lacked the ability to answer specific scientific questions. While various methods have been proposed for ML-physics integration, an important underlying theme -- differentiable modeling -- is not sufficiently recognized. Here we outline the concepts, applicability, and significance of differentiable geoscientific modeling (DG). "Differentiable" refers to accurately and efficiently calculating gradients with respect to model variables, critically enabling the learning of high-dimensional unknown relationships. DG refers to a range of methods connecting varying amounts of prior knowledge to neural networks and training them together, capturing a different scope than physics-guided machine learning and emphasizing first principles. Preliminary evidence suggests DG offers better interpretability and causality than ML, improved generalizability and extrapolation capability, and strong potential for knowledge discovery, while approaching the performance of purely data-driven ML. DG models require less training data while scaling favorably in performance and efficiency with increasing amounts of data. With DG, geoscientists may be better able to frame and investigate questions, test hypotheses, and discover unrecognized linkages.

References in corpus (32)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Methods for Interpreting and Understanding Deep Neural Networks
• DeepXDE: A deep learning library for solving differential equations
• Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A trans-disciplinary review of deep learning research for water resources scientists
• Deep learning to represent sub-grid processes in climate models
• Machine learning accelerated computational fluid dynamics
• Crop Yield Prediction Using Deep Neural Networks
• Enhancing streamflow forecast and extracting insights using long-short term memory networks with data integration at continental scales
• Stable Architectures for Deep Neural Networks
• Physics-Informed Neural Networks for Multiphysics Data Assimilation with Application to Subsurface Transport
• Deep Learning of Subsurface Flow via Theory-guided Neural Network
• Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability
• From calibration to parameter learning: Harnessing the scaling effects of big data in geoscientific modeling
• Differentiable, learnable, regionalized process-based models with physical outputs can approach state-of-the-art hydrologic prediction accuracy
• Prolongation of SMAP to Spatio-temporally Seamless Coverage of Continental US Using a Deep Learning Neural Network
• Automated derivation of the adjoint of high-level transient finite element programs
• The data synergy effects of time-series deep learning models in hydrology
• Learning Constitutive Relations from Indirect Observations Using Deep Neural Networks
• Prediction in ungauged regions with sparse flow duration curves and input-selection ensemble modeling
• Physics guided machine learning using simplified theories
• A Differentiable Programming System to Bridge Machine Learning and Scientific Computing
• Hybrid FEM-NN models: Combining artificial neural networks with the finite element method
• The Need for Ethical, Responsible, and Trustworthy Artificial Intelligence for Environmental Sciences
• Evaluating aleatoric and epistemic uncertainties of time series deep learning models for soil moisture predictions
• Universal Differential Equations for Scientific Machine Learning
• PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
• Variational encoder geostatistical analysis (VEGAS) with an application to large scale riverine bathymetry
• DeepClouds.ai: Deep learning enabled computationally cheap direct numerical simulations
• Bathymetry Inversion using a Deep-Learning-Based Surrogate for Shallow Water Equations Solvers
• Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinity

Cited by in corpus (6)

• When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System
• Interpretable structural model error discovery from sparse assimilation increments using spectral bias-reduced neural networks: A quasi-geostrophic turbulence test case
• Scientific machine learning in Hydrology: a unified perspective
• Scientific Machine Learning of Flow Resistance Using Universal Shallow Water Equations with Differentiable Programming
• A Comparison of Mesh-Free Differentiable Programming and Data-Driven Strategies for Optimal Control under PDE Constraints
• Neural-Parameterized Cellular Automata for Wildfire Spread