A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

Abstract

A key element of understanding the efficacy of overparameterized neural networks is characterizing how they represent functions as the number of weights in the network approaches infinity. In this paper, we characterize the norm required to realize a function $f:{\mathbb{R}}^d\to \mathbb{R}$ as a single hidden-layer ReLU network with an unbounded number of units (infinite width), but where the Euclidean norm of the weights is bounded, including precisely characterizing which functions can be realized with finite norm. This was settled for univariate univariate functions in Savarese et al. (2019), where it was shown that the required norm is determined by the L1-norm of the second derivative of the function. We extend the characterization to multivariate functions (i.e., networks with d input units), relating the required norm to the L1-norm of the Radon transform of a (d+1)/2-power Laplacian of the function. This characterization allows us to show that all functions in Sobolev spaces $W^{s,1}(\mathbb{R})$, $s\ge d+1$, can be represented with bounded norm, to calculate the required norm for several specific functions, and to obtain a depth separation result. These results have important implications for understanding generalization performance and the distinction between neural networks and more traditional kernel learning.

References in corpus (5)

• Understanding deep learning requires rethinking generalization
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Why Deep Neural Networks for Function Approximation?
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Cited by in corpus (8)

• The Role of Neural Network Activation Functions
• Neural tangent kernels, transportation mappings, and universal approximation
• On Sparsity in Overparametrised Shallow ReLU Networks
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
• Depth separation beyond radial functions
• Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks