Abstraction Mechanisms Predict Generalization in Deep Neural Networks

Abstract

A longstanding problem for Deep Neural Networks (DNNs) is understanding their puzzling ability to generalize well. We approach this problem through the unconventional angle of \textit{cognitive abstraction mechanisms}, drawing inspiration from recent neuroscience work, allowing us to define the Cognitive Neural Activation metric (CNA) for DNNs, which is the correlation between information complexity (entropy) of given input and the concentration of higher activation values in deeper layers of the network. The CNA is highly predictive of generalization ability, outperforming norm-and-margin-based generalization metrics on an extensive evaluation of over 100 dataset-and-network-architecture combinations, especially in cases where additive noise is present and/or training labels are corrupted. These strong empirical results show the usefulness of CNA as a generalization metric, and encourage further research on the connection between information complexity and representations in the deeper layers of networks in order to better understand the generalization capabilities of DNNs.

References in corpus (16)

• Learning deep representations by mutual information estimation and maximization
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• A mathematical theory of semantic development in deep neural networks
• On the importance of single directions for generalization
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• The loss surface of deep and wide neural networks
• Shannon Information and Kolmogorov Complexity
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• On the Implicit Bias of Dropout
• Deeper Interpretability of Deep Networks