Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Abstract

In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.

arXiv comment2nd Symposium on Advances in Approximate Bayesian Inference, 2019

References in corpus (4)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Where is the Information in a Deep Neural Network?

Cited by in corpus (4)

• On Information Plane Analyses of Neural Network Classifiers -- A Review
• On the infinite width limit of neural networks with a standard parameterization
• Estimating informativeness of samples with Smooth Unique Information
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization