Universal Approximation with Deep Narrow Networks

Abstract

The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let $n$ be the number of inputs neurons, $m$ be the number of output neurons, and let $\rho$ be any nonaffine continuous function, with a continuous nonzero derivative at some point. Then we show that the class of neural networks of arbitrary depth, width $n+m+2$, and activation function $\rho$, is dense in $C(K;{\mathbb{R}}^m)$ for $K\subseteq {\mathbb{R}}^n$ with $K$ compact. This covers every activation function possible to use in practice, and also includes polynomial activation functions, which is unlike the classical version of the theorem, and provides a qualitative difference between deep narrow networks and shallow wide networks. We then consider several extensions of this result. In particular we consider nowhere differentiable activation functions, density in noncompact domains with respect to the $L^p$-norm, and how the width may be reduced to just $n+m+1$ for `most' activation functions.

Accepted at COLT 2020

References in corpus (9)

• Empirical Evaluation of Rectified Activations in Convolutional Network
• Searching for Activation Functions
• Residual Flows for Invertible Generative Modeling
• ResNet with one-neuron hidden layers is a Universal Approximator
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Nonlinear Approximation and (Deep) ReLU Networks
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks
• How degenerate is the parametrization of neural networks with the ReLU activation function?

Cited by in corpus (8)

• Review: Deep Learning in Electron Microscopy
• The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
• DeepOPF: A Deep Neural Network Approach for Security-Constrained DC Optimal Power Flow
• Interval Universal Approximation for Neural Networks
• Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks
• Improve Adversarial Robustness via Weight Penalization on Classification Layer
• Approximation with Neural Networks in Variable Lebesgue Spaces
• An Informal Introduction to Multiplet Neural Networks