On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

Abstract

We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

References in corpus (11)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• A mean-field limit for certain deep neural networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Representation formulas and pointwise properties for Barron functions
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective

Cited by in corpus (11)

• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• The Modern Mathematics of Deep Learning
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Some observations on high-dimensional partial differential equations with Barron data
• Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• A priori estimates for classification problems using neural networks
• A Dynamical Central Limit Theorem for Shallow Neural Networks
• Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks