On Explicit Super-Expressive Approximation for Neural Networks

Abstract

In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error. We resolve this issue by introducing the Chinese Remainder Theorem as a constructive encoding mechanism. For Lipschitz continuous functions on $[0,1{]}^D$, we construct a width-$\max \{D,4\}$, depth-$5$ network with explicit parameter-error trade-offs. For Hölder-smooth functions in $C_A^{r,{\hat{I}}^3}\left([0,1{]}^D\right)$, our fixed network of width $\max \{2D,D+5N+1\}$ and depth $r+9$ achieves the parameter magnitude $\mathcal{P}$ bounded by ${\log }_2\mathcal{P}=\mathcal{O}({\epsilon }^{-2D/(r+{\hat{I}}^3)}\log (1/\epsilon ))$. This is the dual result compared to those in the parameter-bounded and architecture-unbounded paradigm.

44 pages, 4 figures