How do infinite width bounded norm networks look in function space?

Abstract

We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function. For functions $f:\mathbb{R}\to \mathbb{R}$ and a single hidden layer, we show that the minimal network norm for representing $f$ is $\max (\int |f^{\prime \prime }(x)|dx,|f^{\prime }(-\infty )+f^{\prime }(+\infty )|)$, and hence the minimal norm fit for a sample is given by a linear spline interpolation.

References in corpus (2)

• Understanding deep learning requires rethinking generalization
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate

Cited by in corpus (10)

• Gradient Dynamics of Shallow Univariate ReLU Networks
• Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
• Memorization in Overparameterized Autoencoders
• Approximation power of random neural networks
• Implicit Rugosity Regularization via Data Augmentation
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
• Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
• On Alignment in Deep Linear Neural Networks