Approximation Properties of Deep ReLU CNNs

Abstract

This paper focuses on establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with a large spatial size and multi-channels. Given the decomposition result, the property of the ReLU activation function, and a specific structure for channels, a universal approximation theorem of deep ReLU CNNs with classic structure is obtained by showing its connection with one-hidden-layer ReLU neural networks (NNs). Furthermore, approximation properties are obtained for one version of neural networks with ResNet, pre-act ResNet, and MgNet architecture based on connections between these networks.

30 pages

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The Finite Neuron Method and Convergence Analysis
• ReLU Deep Neural Networks from the Hierarchical Basis Perspective
• Statistical theory for image classification using deep convolutional neural networks with cross-entropy loss under the hierarchical max-pooling model
• Characterization of the Variation Spaces Corresponding to Shallow Neural Networks
• Universal Approximation Theorem for Equivariant Maps by Group CNNs

Cited by in corpus (1)

• Balanced Group Convolution: An Improved Group Convolution Based on Approximability Estimates