The phase diagram of approximation rates for deep neural networks

Abstract

We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results. In particular, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Moreover, we show that all networks with a piecewise polynomial activation function have the same phase diagram. Next, we demonstrate that standard fully-connected architectures with a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we consider deep networks with periodic activations ("deep Fourier expansion") and prove that they have very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.

The final version published at NeurIPS 2020

Cited by in corpus (14)

• Deep Network Approximation for Smooth Functions
• Neural Network Approximation: Three Hidden Layers Are Enough
• The Modern Mathematics of Deep Learning
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• On the capacity of deep generative networks for approximating distributions
• An error analysis of generative adversarial networks for learning distributions
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Approximation in shift-invariant spaces with deep ReLU neural networks
• On Deep Instrumental Variables Estimate
• Reproducing Activation Function for Deep Learning
• Solving PDEs on Unknown Manifolds with Machine Learning
• Stationary Density Estimation of Itô Diffusions Using Deep Learning