Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depth

Abstract

We prove sharp dimension-free representation results for neural networks with $D$ ReLU layers under square loss for a class of functions ${\mathcal{G}}_D$ defined in the paper. These results capture the precise benefits of depth in the following sense: 1. The rates for representing the class of functions ${\mathcal{G}}_D$ via $D$ ReLU layers is sharp up to constants, as shown by matching lower bounds. 2. For each $D$, ${\mathcal{G}}_D\subseteq {\mathcal{G}}_{D+1}$ and as $D$ grows the class of functions ${\mathcal{G}}_D$ contains progressively less smooth functions. 3. If $D^{\prime }<D$, then the approximation rate for the class ${\mathcal{G}}_D$ achieved by depth $D^{\prime }$ networks is strictly worse than that achieved by depth $D$ networks. This constitutes a fine-grained characterization of the representation power of feedforward networks of arbitrary depth $D$ and number of neurons $N$, in contrast to existing representation results which either require $D$ growing quickly with $N$ or assume that the function being represented is highly smooth. In the latter case similar rates can be obtained with a single nonlinear layer. Our results confirm the prevailing hypothesis that deeper networks are better at representing less smooth functions, and indeed, the main technical novelty is to fully exploit the fact that deep networks can produce highly oscillatory functions with few activation functions.

arXiv comment12 pages, 1 figure (surprisingly short isn't it?)

References in corpus (8)

• Optimal approximation of continuous functions by very deep ReLU networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Deep ReLU network approximation of functions on a manifold
• Depth Separation for Neural Networks
• A Corrective View of Neural Networks: Representation, Memorization and Learning
• Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems

Cited by in corpus (6)

• A Corrective View of Neural Networks: Representation, Memorization and Learning
• On the Representation of Solutions to Elliptic PDEs in Barron Spaces
• Size and Depth Separation in Approximating Benign Functions with Neural Networks
• On the Optimal Memorization Power of ReLU Neural Networks
• Depth separation beyond radial functions
• Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem