The gap between theory and practice in function approximation with deep neural networks
arXiv:2001.07523
Abstract
Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. Driven partly by rapidly-expanding literature on DNN approximation theory showing they can approximate a rich variety of functions, such tools are increasingly being considered for problems in scientific computing. Yet, unlike traditional algorithms in this field, little is known about DNNs from the principles of numerical analysis, e.g., stability, accuracy, computational efficiency and sample complexity. In this paper we introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues. We study performance of DNNs of different widths & depths on test functions in various dimensions, including smooth and piecewise smooth functions. We also compare DL against best-in-class methods for smooth function approx. based on compressed sensing (CS). Our main conclusion from these experiments is that there is a crucial gap between the approximation theory of DNNs and their practical performance, with trained DNNs performing relatively poorly on functions for which there are strong approximation results (e.g. smooth functions), yet performing well in comparison to best-in-class methods for other functions. To analyze this gap further, we provide some theoretical insights. We establish a practical existence theorem, asserting existence of a DNN architecture and training procedure that offers the same performance as CS. This establishes a key theoretical benchmark, showing the gap can be closed, albeit via a strategy guaranteed to perform as well as, but no better than, current best-in-class schemes. Nevertheless, it demonstrates the promise of practical DNN approx., by highlighting potential for better schemes through careful design of DNN architectures and training strategies.
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Decoupled Weight Decay Regularization
- Intriguing properties of neural networks
- Machine learning phases of matter
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
- On instabilities of deep learning in image reconstruction - Does AI come at a cost?
- A non-adapted sparse approximation of PDEs with stochastic inputs
- Deep Network Approximation for Smooth Functions
- A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
- Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
- Optimal approximation of continuous functions by very deep ReLU networks
- Polynomial approximation via compressed sensing of high-dimensional functions on lower sets
- DeepFool: a simple and accurate method to fool deep neural networks
- Universal adversarial perturbations
- Full error analysis for the training of deep neural networks
- Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint
- Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
- Neural network integral representations with the ReLU activation function
- Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation rates
- A Polynomial-Based Approach for Architectural Design and Learning with Deep Neural Networks