Resolution and Relevance Trade-offs in Deep Learning
arXiv:1710.11324
Abstract
Deep learning has been successfully applied to various tasks, but its underlying mechanism remains unclear. Neural networks associate similar inputs in the visible layer to the same state of hidden variables in deep layers. The fraction of inputs that are associated to the same state is a natural measure of similarity and is simply related to the cost in bits required to represent these inputs. The degeneracy of states with the same information cost provides instead a natural measure of noise and is simply related the entropy of the frequency of states, that we call relevance. Representations with minimal noise, at a given level of similarity (resolution), are those that maximise the relevance. A signature of such efficient representations is that frequency distributions follow power laws. We show, in extensive numerical experiments, that deep neural networks extract a hierarchy of efficient representations from data, because they i) achieve low levels of noise (i.e. high relevance) and ii) exhibit power law distributions. We also find that the layer that is most efficient to reliably generate patterns of training data is the one for which relevance and resolution are traded at the same price, which implies that frequency distribution follows Zipf's law.
arXiv comment13 pages, 3 figures
References in corpus (12)
- Auto-Encoding Variational Bayes
- Power laws, Pareto distributions and Zipf's law
- Machine learning phases of matter
- Are biological systems poised at criticality?
- Opening the Black Box of Deep Neural Networks via Information
- Accelerate Monte Carlo Simulations with Restricted Boltzmann Machines
- Information-based fitness and the emergence of criticality in living systems
- An exact mapping between the Variational Renormalization Group and Deep Learning
- Zipf's law and criticality in multivariate data without fine-tuning
- Deep Learning with Nonparametric Clustering
- Criticality of mostly informative samples: A Bayesian model selection approach
- Identifying relevant positions in proteins by Critical Variable Selection