Intrinsic dimension of data representations in deep neural networks

Abstract

Deep neural networks progressively transform their inputs across multiple processing layers. What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data-representations, i.e. the minimal number of parameters needed to describe a representation. We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer. Across layers, the ID first increases and then progressively decreases in the final layers. Remarkably, the ID of the last hidden layer predicts classification accuracy on the test set. These results can neither be found by linear dimensionality estimates (e.g., with principal component analysis), nor in representations that had been artificially linearized. They are neither found in untrained networks, nor in networks that are trained on randomized labels. This suggests that neural networks that can generalize are those that transform the data into low-dimensional, but not necessarily flat manifolds.

arXiv commentNeurIPS 2019

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Opening the Black Box of Deep Neural Networks via Information
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information
• Dimensionality-Driven Learning with Noisy Labels
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Classification and Geometry of General Perceptual Manifolds
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Linear Readout of Object Manifolds
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Efficient Representation of Low-Dimensional Manifolds using Deep Networks

Cited by in corpus (29)

• The Gaussian equivalence of generative models for learning with shallow neural networks
• Generative Capacity of Probabilistic Protein Sequence Models
• DADApy: Distance-based Analysis of DAta-manifolds in Python
• Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• What Do Neural Networks Learn When Trained With Random Labels?
• Statistical learning theory of structured data
• Quantifying Relevance in Learning and Inference
• Dataset Distillation with Infinitely Wide Convolutional Networks
• A Neural Scaling Law from the Dimension of the Data Manifold
• Data-driven modelling of brain activity using neural networks, Diffusion Maps, and the Koopman operator
• Improved decision making with similarity based machine learning: Applications in chemistry
• The Geometry of Concepts: Sparse Autoencoder Feature Structure
• Blessing of dimensionality at the edge
• The Intrinsic Manifolds of Radiological Images and their Role in Deep Learning
• Classification Logit Two-sample Testing by Neural Networks
• Probing the Purview of Neural Networks via Gradient Analysis
• On the interplay between data structure and loss function in classification problems
• A random energy approach to deep learning
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• A finite sample analysis of the benign overfitting phenomenon for ridge function estimation
• A Geometric Perspective for High-Dimensional Multiplex Graphs
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamics
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Maximal Relevance and Optimal Learning Machines
• On Data-centric Myths