Banach Space Representer Theorems for Neural Networks and Ridge Splines

Abstract

We develop a variational framework to understand the properties of the functions learned by neural networks fit to data. We propose and study a family of continuous-domain linear inverse problems with total variation-like regularization in the Radon domain subject to data fitting constraints. We derive a representer theorem showing that finite-width, single-hidden layer neural networks are solutions to these inverse problems. We draw on many techniques from variational spline theory and so we propose the notion of polynomial ridge splines, which correspond to single-hidden layer neural networks with truncated power functions as the activation function. The representer theorem is reminiscent of the classical reproducing kernel Hilbert space representer theorem, but we show that the neural network problem is posed over a non-Hilbertian Banach space. While the learning problems are posed in the continuous-domain, similar to kernel methods, the problems can be recast as finite-dimensional neural network training problems. These neural network training problems have regularizers which are related to the well-known weight decay and path-norm regularizers. Thus, our result gives insight into functional characteristics of trained neural networks and also into the design neural network regularizers. We also show that these regularizers promote neural network solutions with desirable generalization properties.

arXiv commentupdate to published version

References in corpus (11)

• Understanding deep learning requires rethinking generalization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Provable approximation properties for deep neural networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• The Role of Neural Network Activation Functions
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Native Banach spaces for splines and variational inverse problems
• Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ${\ell }^1$ and ${\ell }^0$ Controls

Cited by in corpus (8)

• The Role of Neural Network Activation Functions
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Weighted variation spaces and approximation by shallow ReLU networks
• MARS via LASSO
• Embedding Inequalities for Barron-type Spaces
• Ridgeless Interpolation with Shallow ReLU Networks in $1D$ is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions