Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

Abstract

One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.

References in corpus (37)

• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Machine learning and the physical sciences
• Turbulence Modeling in the Age of Data
• Fourier Neural Operator for Parametric Partial Differential Equations
• Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
• Enhancing Computational Fluid Dynamics with Machine Learning
• Learning data driven discretizations for partial differential equations
• Learning to Simulate Complex Physics with Graph Networks
• Deep Fluids: A Generative Network for Parameterized Fluid Simulations
• Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations
• DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks
• Constraint-Aware Neural Networks for Riemann Problems
• Meaningless comparisons lead to false optimism in medical machine learning
• Learned discretizations for passive scalar advection in a 2-D turbulent flow
• Learned Turbulence Modelling with Differentiable Fluid Solvers: Physics-based Loss-functions and Optimisation Horizons
• Attention-Enhanced Neural Network Models for Turbulence Simulation
• Residual-based physics-informed transfer learning: A hybrid method for accelerating long-term CFD simulations via deep learning
• On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements
• Poisson CNN: Convolutional neural networks for the solution of the Poisson equation on a Cartesian mesh
• Enhancement of shock-capturing methods via machine learning
• Teaching the Incompressible Navier-Stokes Equations to Fast Neural Surrogate Models in 3D
• Optimizing a DIscrete Loss (ODIL) to solve forward and inverse problems for partial differential equations using machine learning tools
• A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines
• Numerical investigation of minimum drag profiles in laminar flow using deep learning surrogates
• REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
• Accelerating high order discontinuous Galerkin solvers using neural networks: 3D compressible Navier-Stokes equations
• Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations
• REFORMS: Reporting Standards for Machine Learning Based Science
• Learning to correct spectral methods for simulating turbulent flows
• LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Data
• Blending Neural Operators and Relaxation Methods in PDE Numerical Solvers
• Black Hole Weather Forecasting with Deep Learning: A Pilot Study
• Deep Neural Networks to Correct Sub-Precision Errors in CFD
• A composable autoencoder-based iterative algorithm for accelerating numerical simulations
• The first AI simulation of a black hole
• DS-GPS : A Deep Statistical Graph Poisson Solver (for faster CFD simulations)
• Semi-Implicit Neural Solver for Time-dependent Partial Differential Equations

Cited by in corpus (6)

• Can AI weather models predict out-of-distribution gray swan tropical cyclones?
• Tensor-decomposition-based A Priori Surrogate (TAPS) modeling for ultra large-scale simulations
• Neural Quantum Propagators for Driven-Dissipative Quantum Dynamics
• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates
• Fast approximate solvers for metamaterials design in electromagnetism