The Intrinsic Dimension of Images and Its Impact on Learning

Abstract

It is widely believed that natural image data exhibits low-dimensional structure despite the high dimensionality of conventional pixel representations. This idea underlies a common intuition for the remarkable success of deep learning in computer vision. In this work, we apply dimension estimation tools to popular datasets and investigate the role of low-dimensional structure in deep learning. We find that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images. Additionally, we find that low dimensional datasets are easier for neural networks to learn, and models solving these tasks generalize better from training to test data. Along the way, we develop a technique for validating our dimension estimation tools on synthetic data generated by GANs allowing us to actively manipulate the intrinsic dimension by controlling the image generation process. Code for our experiments may be found here https://github.com/ppope/dimensions.

To appear at ICLR 2021 (spotlight), 17 pages with appendix, 15 figures

Cited by in corpus (6)

• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Faster Algorithms for Fair Max-Min Diversification in ${\mathbb{R}}^d$
• On the interplay between data structure and loss function in classification problems
• Towards Empirical Sandwich Bounds on the Rate-Distortion Function
• Deep Networks Provably Classify Data on Curves
• Learning Curves for SGD on Structured Features