Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Abstract

This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple, computable, and continuous activation function $\sigma$ leveraging a triangular-wave function and the softsign function. We first prove that $\sigma$-activated networks with width $36d(2d+1)$ and depth $11$ can approximate any continuous function on a $d$-dimensional hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than $36d(2d+1)\times 11$ is dense in the continuous function space $C([a,b{]}^d)$ and therefore dense in the Lebesgue spaces $L^p([a,b{]}^d)$ for $p\in [1,\infty )$. Furthermore, we show that classification functions arising from image and signal classification are in the hypothesis space generated by $\sigma$-activated networks with width $36d(2d+1)$ and depth $12$ when there exist pairwise disjoint bounded closed subsets of ${\mathbb{R}}^d$ such that the samples of the same class are located in the same subset. Finally, we use numerical experimentation to show that replacing the rectified linear unit (ReLU) activation function by ours would improve the experiment results.

References in corpus (19)

• Deep Network Approximation for Smooth Functions
• The loss surface of deep and wide neural networks
• ResNet with one-neuron hidden layers is a Universal Approximator
• Optimal approximation of continuous functions by very deep ReLU networks
• Neural Network Approximation: Three Hidden Layers Are Enough
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Nonlinear Approximation and (Deep) ReLU Networks
• DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• An overview on deep learning-based approximation methods for partial differential equations
• A Neural Network based Shock Detection and Localization Approach for Discontinuous Galerkin Methods
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• The phase diagram of approximation rates for deep neural networks
• A Priori Estimates of the Population Risk for Residual Networks
• Compositional Distributional Semantics with Long Short Term Memory
• Representation formulas and pointwise properties for Barron functions
• Approximation in shift-invariant spaces with deep ReLU neural networks
• A priori estimates for classification problems using neural networks

Cited by in corpus (3)

• Noncompact uniform universal approximation
• Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
• Don't Fear Peculiar Activation Functions: EUAF and Beyond