Information Losses in Neural Classifiers from Sampling
arXiv:1902.05991 · doi:10.1109/TNNLS.2019.2952029
Abstract
This paper considers the subject of information losses arising from the finite datasets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled datasets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. The paper then uses these bounds to explain some recent experimental findings of information compression in neural networks which cannot be explained by previous work. Finally, the paper shows that not only are these bounds much smaller than existing ones, but that they also correspond well with experiments.
To be published in IEEE TNNLS
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- The information bottleneck method
- OpenML: networked science in machine learning
- Opening the Black Box of Deep Neural Networks via Information
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
- MINE: Mutual Information Neural Estimation
- Train faster, generalize better: Stability of stochastic gradient descent
- Auxiliary Deep Generative Models
- Hierarchical Implicit Models and Likelihood-Free Variational Inference
- High-Dimensional Probability Estimation with Deep Density Models
- Entropy Bounds for Discrete Random Variables via Maximal Coupling