Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
arXiv:2012.06081
Abstract
Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. This arises in science and engineering problems, in particular those involving solution of parametric Partial Differential Equations (PDEs). Such problems are challenging: 1) pointwise samples are expensive to acquire, 2) the function domain is high dimensional, and 3) the range lies in a Hilbert space. Our contributions are twofold. First, we present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. This result introduces a DNN training procedure and full theoretical analysis with explicit guarantees on error and sample complexity. The error bound is explicit in three key errors occurring in the approximation procedure: the best approximation, measurement, and physical discretization errors. Our result shows that there exists a procedure (albeit non-standard) for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes. It gives a benchmark lower bound for how well DNNs can perform on such problems. Second, we examine whether better performance can be achieved in practice through different types of architectures and training. We provide preliminary numerical results illustrating practical performance of DNNs on parametric PDEs. We consider different parameters, modifying the DNN architecture to achieve better and competitive results, comparing these to current best-in-class schemes.
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- DOLFIN: Automated Finite Element Computing
- Machine learning prediction errors better than DFT accuracy
- A unified deep artificial neural network approach to partial differential equations in complex geometries
- A non-adapted sparse approximation of PDEs with stochastic inputs
- Compressive Sampling of Polynomial Chaos Expansions: Convergence Analysis and Sampling Strategies
- Deep Network Approximation for Smooth Functions
- A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
- Optimal approximation of continuous functions by very deep ReLU networks
- Polynomial approximation via compressed sensing of high-dimensional functions on lower sets
- Data driven approximation of parametrized PDEs by Reduced Basis and Neural Networks
- Full error analysis for the training of deep neural networks
- Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
- Deep Learning Techniques for Inverse Problems in Imaging
- Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation rates
- A Polynomial-Based Approach for Architectural Design and Learning with Deep Neural Networks