Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

Abstract

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in $W^{2,\infty }((a,b{)}^d)$ can be approximated with arbitrary accuracy, measured in the $W^{1,\infty }$-norm, by a fixed-size neural network using the Elementary Universal Activation Function ($\mathrm{EUAF}$). To extend this result to $W^{s,\infty }((a,b{)}^d)$ for $s\in \mathbb{N}$, we introduce a smooth activation ${\mathrm{DUAF}}_{\infty }$ from the family of Differentiable Universal Activation Functions (${\mathrm{DUAF}}_n$). We prove that any function in $W^{s,\infty }((a,b{)}^d)$ can be approximated with arbitrary accuracy in the $W^{s-1,\infty }$-norm by a fixed-size ${\mathrm{DUAF}}_{\infty }$-activated network. We further construct sigmoidal variants ${\widetilde{\mathrm{DUAF}}}_n$ and show that, for every $1\le s\le n$, fixed-size ${\widetilde{\mathrm{DUAF}}}_n$-activated networks still approximate any $f\in W^{s,\infty }((a,b{)}^d)$ with arbitrary accuracy in the $W^{s-1,\infty }$-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.