CNNs Avoid Curse of Dimensionality by Learning on Patches
arXiv:2205.10760 · doi:10.1109/OJSP.2023.3270082
Abstract
Despite the success of convolutional neural networks (CNNs) in numerous computer vision tasks and their extraordinary generalization performances, several attempts to predict the generalization errors of CNNs have only been limited to a posteriori analyses thus far. A priori theories explaining the generalization performances of deep neural networks have mostly ignored the convolutionality aspect and do not specify why CNNs are able to seemingly overcome curse of dimensionality on computer vision tasks like image classification where the image dimensions are in thousands. Our work attempts to explain the generalization performance of CNNs on image classification under the hypothesis that CNNs operate on the domain of image patches. Ours is the first work we are aware of to derive an a priori error bound for the generalization error of CNNs and we present both quantitative and qualitative evidences in the support of our theory. Our patch-based theory also offers explanation for why data augmentation techniques like Cutout, CutMix and random cropping are effective in improving the generalization error of CNNs.
arXiv commentReorganization
References in corpus (17)
- Explaining and Harnessing Adversarial Examples
- Intriguing properties of neural networks
- mixup: Beyond Empirical Risk Minimization
- Improved Regularization of Convolutional Neural Networks with Cutout
- Scaling Laws for Neural Language Models
- Training Compute-Optimal Large Language Models
- SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation
- The Power of Depth for Feedforward Neural Networks
- Fantastic Generalization Measures and Where to Find Them
- On the Expressive Power of Deep Learning: A Tensor Analysis
- Representation Benefits of Deep Feedforward Networks
- Norm-Based Capacity Control in Neural Networks
- Dimensionality compression and expansion in Deep Neural Networks
- Generalization Error of Invariant Classifiers
- Understanding Generalization and Optimization Performance of Deep CNNs
- Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
- Generalization Bounds for Convolutional Neural Networks