Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow

Abstract

Transfer learning (TL) is becoming a powerful tool in scientific applications of neural networks (NNs), such as weather/climate prediction and turbulence modeling. TL enables out-of-distribution generalization (e.g., extrapolation in parameters) and effective blending of disparate training sets (e.g., simulations and observations). In TL, selected layers of a NN, already trained for a base system, are re-trained using a small dataset from a target system. For effective TL, we need to know 1) what are the best layers to re-train? and 2) what physics are learned during TL? Here, we present novel analyses and a new framework to address (1)-(2) for a broad range of multi-scale, nonlinear systems. Our approach combines spectral analyses of the systems' data with spectral analyses of convolutional NN's activations and kernels, explaining the inner-workings of TL in terms of the system's nonlinear physics. Using subgrid-scale modeling of several setups of 2D turbulence as test cases, we show that the learned kernels are combinations of low-, band-, and high-pass filters, and that TL learns new filters whose nature is consistent with the spectral differences of base and target systems. We also find the shallowest layers are the best to re-train in these cases, which is against the common wisdom guiding TL in machine learning literature. Our framework identifies the best layer(s) to re-train beforehand, based on physics and NN theory. Together, these analyses explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.

21 pages, 6 figures

References in corpus (6)

• Deep transfer operator learning for partial differential equations under conditional shift
• Data-driven subgrid-scale modeling of forced Burgers turbulence using deep learning with generalization to higher Reynolds numbers via transfer learning
• A posteriori learning for quasi-geostrophic turbulence parametrization
• Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks
• Transfer learning for nonlinear dynamics and its application to fluid turbulence
• Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks

Cited by in corpus (11)

• In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges
• Explainable Offline-Online Training of Neural Networks for Parameterizations: A 1D Gravity Wave-QBO Testbed in the Small-data Regime
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Transferring climate change physical knowledge
• Interpretable structural model error discovery from sparse assimilation increments using spectral bias-reduced neural networks: A quasi-geostrophic turbulence test case
• Reduced Data-Driven Turbulence Closure for Capturing Long-Term Statistics
• Advancing global sea ice prediction capabilities using a fully-coupled climate model with integrated machine learning
• A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields
• Electron neural closure for turbulent magnetosheath simulations: energy channels
• Guided Unconditional and Conditional Generative Models for Super-Resolution and Inference of Quasi-Geostrophic Turbulence