Approximation bounds for norm constrained neural networks with applications to regression and GANs

Abstract

This paper studies the approximation capacity of ReLU neural networks with norm constraint on the weights. We prove upper and lower bounds on the approximation error of these networks for smooth function classes. The lower bound is derived through the Rademacher complexity of neural networks, which may be of independent interest. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs. In particular, we obtain convergence rates for over-parameterized neural networks. It is also shown that GANs can achieve optimal rate of learning probability distributions, when the discriminator is a properly chosen norm constrained neural network.

References in corpus (7)

• Deep Network Approximation for Smooth Functions
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Optimal approximation of continuous functions by very deep ReLU networks
• Deep Network Approximation Characterized by Number of Neurons
• On the capacity of deep generative networks for approximating distributions
• An error analysis of generative adversarial networks for learning distributions
• Approximation in shift-invariant spaces with deep ReLU neural networks

Cited by in corpus (2)

• Optimal rates of approximation by shallow ReLU${}^k$ neural networks and applications to nonparametric regression
• On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks