Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

Qi Zhou; Xuan Zhou; Xiao-Song Yang

machine learning

Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

arXiv:2607.04597

summary

The paper analyzes the minimal block width required for residual neural networks with inner width one to achieve universal function approximation, proving it equals the larger of input or output dimensions and providing bounds for both L^p and uniform approximation.

Abstract

In this paper, we study the universal approximation property of residual neural networks. For input and output dimensions $d_{x}$ and $d_{y}$ , and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve $L^{p}$ approximation $(1 \leq p < + \infty)$ on any compact set, we show that the exact minimum block width is $max {d_{x}, d_{y}}$ when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width $min {d_{x} + d_{y}, max {2 d_{x} + 1, d_{y}}}$ can achieve uniform approximation on any compact set under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than $max {d_{x}, d_{y}}$ , both in the $L^{p}$ sense and the uniform sense, regardless of inner width. Consequently, for LeakyReLU, ReLU, ReLU-like activation functions and $d_{y} \geq 2 d_{x} + 1$ , the exact minimum block width for uniform approximation is $d_{y}$ when each residual branch has inner width 1.

Topics & keywords

#universal approximation #residual networks #network width #function approximation #deep learning theoryresidual neural networkblock widthinner widthLeakyReLUReLUL^p approximationuniform approximation