Provable approximation properties for deep neural networks

Abstract

We discuss approximation of functions using deep neural nets. Given a function $f$ on a $d$-dimensional manifold $\hat{I}“\subset {\mathbb{R}}^m$, we construct a sparsely-connected depth-4 neural network and bound its error in approximating $f$. The size of the network depends on dimension and curvature of the manifold $\hat{I}“$, the complexity of $f$, in terms of its wavelet description, and only weakly on the ambient dimension $m$. Essentially, our network computes wavelet functions, which are computed from Rectified Linear Units (ReLU)

accepted for publication in Applied and Computational Harmonic Analysis