Representation formulas and pointwise properties for Barron functions

Abstract

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $C^1$-diffeomorphisms which Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.

References in corpus (3)

• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Some observations on high-dimensional partial differential equations with Barron data
• A Note on the Representation Power of GHHs

Cited by in corpus (12)

• The Modern Mathematics of Deep Learning
• The Finite Neuron Method and Convergence Analysis
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Some observations on high-dimensional partial differential equations with Barron data
• Characterization of the Variation Spaces Corresponding to Shallow Neural Networks
• A priori estimates for classification problems using neural networks
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Sharp Lower Bounds on the Approximation Rate of Shallow Neural Networks
• Tighter Sparse Approximation Bounds for ReLU Neural Networks