Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow

Abstract

Transfer learning (TL) is becoming a powerful tool in scientific applications of neural networks (NNs), such as weather/climate prediction and turbulence modeling. TL enables out-of-distribution generalization (e.g., extrapolation in parameters) and effective blending of disparate training sets (e.g., simulations and observations). In TL, selected layers of a NN, already trained for a base system, are re-trained using a small dataset from a target system. For effective TL, we need to know 1) what are the best layers to re-train? and 2) what physics are learned during TL? Here, we present novel analyses and a new framework to address (1)-(2) for a broad range of multi-scale, nonlinear systems. Our approach combines spectral analyses of the systems' data with spectral analyses of convolutional NN's activations and kernels, explaining the inner-workings of TL in terms of the system's nonlinear physics. Using subgrid-scale modeling of several setups of 2D turbulence as test cases, we show that the learned kernels are combinations of low-, band-, and high-pass filters, and that TL learns new filters whose nature is consistent with the spectral differences of base and target systems. We also find the shallowest layers are the best to re-train in these cases, which is against the common wisdom guiding TL in machine learning literature. Our framework identifies the best layer(s) to re-train beforehand, based on physics and NN theory. Together, these analyses explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.

arXiv comment21 pages, 6 figures

References in corpus (29)

• Machine Learning for Fluid Mechanics
• Solving high-dimensional partial differential equations using deep learning
• Deep learning to represent sub-grid processes in climate models
• Machine learning accelerated computational fluid dynamics
• Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems
• Deep Neural Networks for Data-Driven Turbulence Models
• Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations
• Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions
• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• Sub-grid modelling for two-dimensional turbulence using neural networks
• On Lazy Training in Differentiable Programming
• Transfer learning based multi-fidelity physics informed deep neural network
• Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for WeatherBench
• Deep transfer operator learning for partial differential equations under conditional shift
• Stable a posteriori LES of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher Re via transfer learning
• Do Adversarially Robust ImageNet Models Transfer Better?
• High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network
• Data-driven subgrid-scale modeling of forced Burgers turbulence using deep learning with generalization to higher Reynolds numbers via transfer learning
• A posteriori learning for quasi-geostrophic turbulence parametrization
• Learning physics-constrained subgrid-scale closures in the small-data regime for stable and accurate LES
• Turbulence closure modeling with data-driven techniques: physical compatibility and consistency considerations
• Data-driven super-parameterization using deep learning: Experimentation with multi-scale Lorenz 96 systems and transfer-learning
• Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks
• Revealing the state space of turbulence using machine learning
• Physical invariance in neural networks for subgrid-scale scalar flux modeling
• Transfer learning for nonlinear dynamics and its application to fluid turbulence
• Interpretable Data-driven Methods for Subgrid-scale Closure in LES for Transcritical LOX/GCH4 Combustion
• Frame invariant neural network closures for Kraichnan turbulence
• Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks

Cited by in corpus (11)

• In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges
• Explainable Offline-Online Training of Neural Networks for Parameterizations: A 1D Gravity Wave-QBO Testbed in the Small-data Regime
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Transferring climate change physical knowledge
• Interpretable structural model error discovery from sparse assimilation increments using spectral bias-reduced neural networks: A quasi-geostrophic turbulence test case
• Reduced Data-Driven Turbulence Closure for Capturing Long-Term Statistics
• Advancing global sea ice prediction capabilities using a fully-coupled climate model with integrated machine learning
• A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields
• Electron neural closure for turbulent magnetosheath simulations: energy channels
• Guided Unconditional and Conditional Generative Models for Super-Resolution and Inference of Quasi-Geostrophic Turbulence