The Universal Approximation Property

Abstract

The universal approximation property of various machine learning models is currently only understood on a case-by-case basis, limiting the rapid development of new theoretically justified neural network architectures and blurring our understanding of our current models' potential. This paper works towards overcoming these challenges by presenting a characterization, a representation, a construction method, and an existence result, each of which applies to any universal approximator on most function spaces of practical interest. Our characterization result is used to describe which activation functions allow the feed-forward architecture to maintain its universal approximation capabilities when multiple constraints are imposed on its final layers and its remaining layers are only sparsely connected. These include a rescaled and shifted Leaky ReLU activation function but not the ReLU activation function. Our construction and representation result is used to exhibit a simple modification of the feed-forward architecture, which can approximate any continuous function with non-pathological growth, uniformly on the entire Euclidean input space. This improves the known capabilities of the feed-forward architecture.

References in corpus (13)

• Self-Attention Generative Adversarial Networks
• Searching for Activation Functions
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
• A generative adversarial network approach to calibration of local stochastic volatility models
• Regression on fixed-rank positive semidefinite matrices: a Riemannian approach
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• Hyperbolic Neural Networks
• Non-Euclidean Universal Approximation
• Dynamics of weighted composition operators on function spaces defined by local properties
• Extending and improving conical bicombings
• Deep Learning in a Generalized HJM-type Framework Through Arbitrage-Free Regularization
• Low-Rank plus Sparse Decomposition of Covariance Matrices using Neural Network Parametrization

Cited by in corpus (9)

• Self-Supervised Nonlinear Transform-Based Tensor Nuclear Norm for Multi-Dimensional Image Recovery
• Continuum-extrapolated NNLO Valence PDF of Pion at the Physical Point
• Heavy quark potential in quark-gluon Plasma: Deep neural network meets lattice quantum chromodynamics
• Towards constraining QCD phase transitions in neutron star interiors: Bayesian Inference with TOV linear response analysis
• Blind Transmitter Localization Using Deep Learning: A Scalability Study
• Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions
• Decoupling multivariate functions using a nonparametric filtered tensor decomposition
• Neural Networks in Fréchet spaces
• (Thesis) Reservoir Computing With Dynamical Systems