Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation

Abstract

To achieve near-zero training error in a classification problem, the layers of a feed-forward network have to disentangle the manifolds of data points with different labels, to facilitate the discrimination. However, excessive class separation can bring to overfitting since good generalisation requires learning invariant features, which involve some level of entanglement. We report on numerical experiments showing how the optimisation dynamics finds representations that balance these opposing tendencies with a non-monotonic trend. After a fast segregation phase, a slower rearrangement (conserved across data sets and architectures) increases the class entanglement.The training error at the inversion is stable under subsampling, and across network initialisations and optimisers, which characterises it as a property solely of the data structure and (very weakly) of the architecture. The inversion is the manifestation of tradeoffs elicited by well-defined and maximally stable elements of the training set, coined ``stragglers'', particularly influential for generalisation.

References in corpus (20)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Reconciling modern machine learning practice and the bias-variance trade-off
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information
• Mean-field message-passing equations in the Hopfield model and its generalizations
• How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks
• Disentangling feature and lazy training in deep neural networks
• Where is the Information in a Deep Neural Network?
• Semi-Supervised Learning via Compact Latent Space Clustering
• Dimensionality compression and expansion in Deep Neural Networks
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Zipf and Heaps laws from dependency structures in component systems
• Data-driven emergence of convolutional structure in neural networks
• Intrinsic dimension estimation for locally undersampled data
• Learning through atypical "phase transitions" in overparameterized neural networks
• Statistical learning theory of structured data
• Spin glass theory and its new challenge: structured disorder
• Counting the learnable functions of structured data
• Beyond the storage capacity: data driven satisfiability transition
• Linear Classification of Neural Manifolds with Correlated Variability