The universal approximation power of finite-width deep ReLU networks

Abstract

We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with their recently established universal approximation property of affine function systems (Bölcskei et al., 2018), this shows that deep neural networks approximate vastly different signal structures generated by the affine group, the Weyl-Heisenberg group, or through warping, and even certain fractals, all with approximation error decaying exponentially in the number of neurons. We also prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly smaller connectivity than finite-depth wide networks.

References in corpus (3)

• Representation Benefits of Deep Feedforward Networks
• On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks
• Error bounds for approximations with deep ReLU networks

Cited by in corpus (6)

• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing
• Full error analysis for the training of deep neural networks
• How degenerate is the parametrization of neural networks with the ReLU activation function?
• Information-Theoretic Lower Bounds for Compressive Sensing with Generative Models
• Finding Everything within Random Binary Networks