Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

Abstract

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous function on the unit cube $[0,1{]}^d$ aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? Our approach to this paper is based on the observation that, due to the convexity of the ReLU activation, ReLU nets are particularly well-suited for representing convex functions. In particular, we prove that ReLU nets with width $d+1$ can approximate any continuous convex function of $d$ variables arbitrarily well. These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the $d$-dimensional cube $[0,1{]}^d$ by ReLU nets with width $d+3.$

v3. Theorem 3 removed. Comments Welcome. 9p

References in corpus (4)

• Why does deep and cheap learning work so well?
• Understanding Deep Neural Networks with Rectified Linear Units
• The power of deeper networks for expressing natural functions
• Neural networks and rational functions

Cited by in corpus (29)

• Deep Neural Networks for Estimation and Inference
• A Review on Deep Learning in Medical Image Reconstruction
• Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning Based Methods
• The Universal Approximation Property
• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression
• Mathematical Models of Overparameterized Neural Networks
• Prediction of numerical homogenization using deep learning for the Richards equation
• Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regression
• Robust learning with implicit residual networks
• Multi-Task Learning for Multi-Dimensional Regression: Application to Luminescence Sensing
• Provably Good Solutions to the Knapsack Problem via Neural Networks of Bounded Size
• Approximation in shift-invariant spaces with deep ReLU neural networks
• Predicting the transverse emittance of space charge dominated beams using the phase advance scan technique and a fully connected neural network
• Reliable Off-policy Evaluation for Reinforcement Learning
• Probing nuclear physics with supernova gravitational waves and machine learning
• Physically interpretable machine learning algorithm on multidimensional non-linear fields
• A Deep Learning Approach Towards Generating High-fidelity Diverse Synthetic Battery Datasets
• Model reduction in acoustic inversion by artificial neural network
• Learning Algebraic Models of Quantum Entanglement
• Towards Lower Bounds on the Depth of ReLU Neural Networks
• Neural Network Predicts Ion Concentration Profiles under Nanoconfinement
• Perspectives for analyzing non-linear photo ionization spectra with deep neural networks trained with synthetic Hamilton matrices
• Analysis of Structured Deep Kernel Networks
• Machine-learning based high-bandwidth magnetic sensing
• NPLIC: A Machine Learning Approach to Piecewise Linear Interface Construction
• ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation
• YANNs: Y-wise Affine Neural Networks for Exact and Efficient Representations of Piecewise Linear Functions
• Scheduling and Power Control for Wireless Multicast Systems via Deep Reinforcement Learning