ResNet with one-neuron hidden layers is a Universal Approximator

Abstract

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. ${\ell }_1({\mathbb{R}}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and $d$. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension $d$ [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.

References in corpus (7)

• The loss surface of deep and wide neural networks
• Optimal approximation of continuous functions by very deep ReLU networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Global optimality conditions for deep neural networks
• Are ResNets Provably Better than Linear Predictors?
• Weight Sharing is Crucial to Succesful Optimization

Cited by in corpus (35)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Real-time Power System State Estimation and Forecasting via Deep Neural Networks
• A Review on Deep Learning in Medical Image Reconstruction
• SpinalNet: Deep Neural Network with Gradual Input
• Review: Deep Learning in Electron Microscopy
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Choose a Transformer: Fourier or Galerkin
• Augmented Neural ODEs
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• On Interpretability of Artificial Neural Networks: A Survey
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems
• Universal Approximation with Deep Narrow Networks
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory
• Interval Universal Approximation for Neural Networks
• On the Modularity of Hypernetworks
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• State Space Representations of Deep Neural Networks
• Understanding the computational demands underlying visual reasoning
• Siamese Graph Neural Networks for Data Integration
• ResNet After All? Neural ODEs and Their Numerical Solution
• Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers
• Universal Approximation with Quadratic Deep Networks
• Are deep ResNets provably better than linear predictors?
• A needle-based deep-neural-network camera
• Optimal Function Approximation with Relu Neural Networks
• On a Sparse Shortcut Topology of Artificial Neural Networks
• On Residual Networks Learning a Perturbation from Identity
• Neural interval-censored survival regression with feature selection
• Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions
• Deep learning in physics: a study of dielectric quasi-cubic particles in a uniform electric field
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy
• Training Deep Neural Networks via Branch-and-Bound