The Role of Information Complexity and Randomization in Representation Learning

Abstract

A grand challenge in representation learning is to learn the different explanatory factors of variation behind the high dimen- sional data. Encoder models are often determined to optimize performance on training data when the real objective is to generalize well to unseen data. Although there is enough numerical evidence suggesting that noise injection (during training) at the representation level might improve the generalization ability of encoders, an information-theoretic understanding of this principle remains elusive. This paper presents a sample-dependent bound on the generalization gap of the cross-entropy loss that scales with the information complexity (IC) of the representations, meaning the mutual information between inputs and their representations. The IC is empirically investigated for standard multi-layer neural networks with SGD on MNIST and CIFAR-10 datasets; the behaviour of the gap and the IC appear to be in direct correlation, suggesting that SGD selects encoders to implicitly minimize the IC. We specialize the IC to study the role of Dropout on the generalization capacity of deep encoders which is shown to be directly related to the encoder capacity, being a measure of the distinguishability among samples from their representations. Our results support some recent regularization methods.

arXiv comment35 pages, 3 figures. Submitted for publication

References in corpus (11)

• Auto-Encoding Variational Bayes
• The information bottleneck method
• Understanding deep learning requires rethinking generalization
• Opening the Black Box of Deep Neural Networks via Information
• Deep Variational Information Bottleneck
• The Power of Depth for Feedforward Neural Networks
• Information-theoretic analysis of generalization capability of learning algorithms
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Norm-Based Capacity Control in Neural Networks
• How much does your data exploration overfit? Controlling bias via information usage
• Deep vs. shallow networks : An approximation theory perspective

Cited by in corpus (3)

• Learning Optimal Representations with the Decodable Information Bottleneck
• Information Bottleneck and its Applications in Deep Learning
• Compressive Visual Representations