Shallow ReLU${}^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Abstract

This paper studies approximation by shallow ReLU${}^s$ networks, ${\sigma }_s(t)=\max \{0,t{\}}^s$, together with their generalization behavior under ${\ell }_1$ path-norm control. For the $L^p$-type integral spaces ${\tilde{\mathcal{F}}}_{p,{\tau }_d,s}$, $1\le p\le 2$, spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when ${\tau }_d$ is the uniform measure and $1\le p<2$, the approximation rate is $O\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right)$ for $1\le p\le p^{\ast }$ and $O\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right)$ for $p^{\ast }<p<2$, where $p^{\ast }=\frac{2d+2}{d+3}$. Approximation bounds for Sobolev spaces $W^{\alpha ,p}$, $1\le p<2$, are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU${}^s$ networks achieve minimax-optimal rates $O\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right)$ over ${\mathcal{B}}_s$ and $O\left(n^{-\frac{2\alpha }{2\alpha +d}}\log n\right)$ over $W^{\alpha ,\infty }$, with matching lower bounds up to logarithmic factors.

arXiv comment42 pages, 1 figure. Update theorem 2and fix some typos. Authors are listed in alphabetical order and contributed equally