Neural networks with superexpressive activations and integer weights

Abstract

An example of an activation function $\sigma$ is given such that networks with activations $\{\sigma ,\lfloor \cdot \rfloor \}$, integer weights and a fixed architecture depending on $d$ approximate continuous functions on $[0,1{]}^d$. The range of integer weights required for $\epsilon$-approximation of Hölder continuous functions is derived, which leads to a convergence rate of order $n^{\frac{-2\beta }{2\beta +d}}{\log }_{2}n$ for neural network regression estimation of unknown $\beta$-Hölder continuous function with given $n$ samples.