Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks

Abstract

Convolutional neural networks (CNNs) have been shown to achieve optimal approximation and estimation error rates (in minimax sense) in several function classes. However, previous analyzed optimal CNNs are unrealistically wide and difficult to obtain via optimization due to sparse constraints in important function classes, including the Hölder class. We show a ResNet-type CNN can attain the minimax optimal error rates in these classes in more plausible situations -- it can be dense, and its width, channel size, and filter size are constant with respect to sample size. The key idea is that we can replicate the learning ability of Fully-connected neural networks (FNNs) by tailored CNNs, as long as the FNNs have \textit{block-sparse} structures. Our theory is general in a sense that we can automatically translate any approximation rate achieved by block-sparse FNNs into that by CNNs. As an application, we derive approximation and estimation error rates of the aformentioned type of CNNs for the Barron and Hölder classes with the same strategy.

Version 4: Fixed the constant B^{(fc)} in Theorems 1, 5 and the norm upper bound of w^{(l)}_m in Lemma 1. 8 pages + References 2 pages + Supplemental material 18 pages

Cited by in corpus (8)

• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• The Modern Mathematics of Deep Learning
• Approximation Properties of Deep ReLU CNNs
• Statistical theory for image classification using deep convolutional neural networks with cross-entropy loss under the hierarchical max-pooling model
• Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks
• Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks
• Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
• Theory of Deep Convolutional Neural Networks II: Spherical Analysis