On the influence of over-parameterization in manifold based surrogates and deep neural operators

Abstract

Constructing accurate and generalizable approximators for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. We demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of input functions and output QoIs. To enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

28 pages, 7 figures

References in corpus (3)

• Operator learning for predicting multiscale bubble growth dynamics
• Data-driven surrogates for high dimensional models using Gaussian process regression on the Grassmann manifold
• Robust Adaptive Least Squares Polynomial Chaos Expansions in High-Frequency Applications

Cited by in corpus (7)

• Separable DeepONet: Breaking the Curse of Dimensionality in Physics-Informed Machine Learning
• SVD Perspectives for Augmenting DeepONet Flexibility and Interpretability
• Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solving
• Real-Time Prediction of Gas Flow Dynamics in Diesel Engines using a Deep Neural Operator Framework
• Polynomial Chaos Expansions on Principal Geodesic Grassmannian Submanifolds for Surrogate Modeling and Uncertainty Quantification
• A Resolution Independent Neural Operator
• Neural Chaos: A Spectral Stochastic Neural Operator