Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Abstract

In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.

2nd Symposium on Advances in Approximate Bayesian Inference, 2019

Cited by in corpus (4)

• On Information Plane Analyses of Neural Network Classifiers -- A Review
• On the infinite width limit of neural networks with a standard parameterization
• Estimating informativeness of samples with Smooth Unique Information
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization