Neural Network Approximation: Three Hidden Layers Are Enough

Abstract

A three-hidden-layer neural network with super approximation power is introduced. This network is built with the floor function ($\lfloor x\rfloor$), the exponential function ($2^x$), the step function ($1_{x\ge 0}$), or their compositions as the activation function in each neuron and hence we call such networks as Floor-Exponential-Step (FLES) networks. For any width hyper-parameter $N\in {\mathbb{N}}^{+}$, it is shown that FLES networks with width $\max \{d,N\}$ and three hidden layers can uniformly approximate a Hölder continuous function $f$ on $[0,1{]}^d$ with an exponential approximation rate $3\lambda (2\sqrt{d}{)}^{\alpha }{2}^{-\alpha N}$, where $\alpha \in (0,1]$ and $\lambda >0$ are the Hölder order and constant, respectively. More generally for an arbitrary continuous function $f$ on $[0,1{]}^d$ with a modulus of continuity ${\omega }_f(\cdot )$, the constructive approximation rate is $2{\omega }_f(2\sqrt{d})2^{-N}+{\omega }_f(2\sqrt{d}{2}^{-N})$. Moreover, we extend such a result to general bounded continuous functions on a bounded set $E\subseteq {\mathbb{R}}^d$. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of ${\omega }_f(r)$ as $r\to 0$ is moderate (e.g., ${\omega }_f(r)\lesssim r^{\alpha }$ for Hölder continuous functions), since the major term to be concerned in our approximation rate is essentially $\sqrt{d}$ times a function of $N$ independent of $d$ within the modulus of continuity. Finally, we extend our analysis to derive similar approximation results in the $L^p$-norm for $p\in [1,\infty )$ via replacing Floor-Exponential-Step activation functions by continuous activation functions.

References in corpus (23)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Deep Network Approximation for Smooth Functions
• A consensus-based model for global optimization and its mean-field limit
• Optimal approximation of continuous functions by very deep ReLU networks
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Deep Network Approximation Characterized by Number of Neurons
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Nonlinear Approximation via Compositions
• Approximation capability of two hidden layer feedforward neural networks with fixed weights
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• The phase diagram of approximation rates for deep neural networks
• Overcoming the curse of dimensionality in the approximative pricing of financial derivatives with default risks
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
• Approximation in shift-invariant spaces with deep ReLU neural networks
• A consensus-based global optimization method for high dimensional machine learning problems

Cited by in corpus (17)

• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Reproducing Activation Function for Deep Learning
• Noncompact uniform universal approximation
• Are Two Hidden Layers Still Enough for the Physics-Informed Neural Networks?
• Solving PDEs on Unknown Manifolds with Machine Learning
• Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks
• Simultaneous approximation by neural network operators with applications to Voronovskaja formulas
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
• Don't Fear Peculiar Activation Functions: EUAF and Beyond
• Why Shallow Networks Struggle to Approximate and Learn High Frequencies
• Elementary superexpressive activations
• Lyapunov-Net: A Deep Neural Network Architecture for Lyapunov Function Approximation
• XNet-Enhanced Deep BSDE Method and Numerical Analysis