Information theory and learning: a physical approach

Abstract

We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series, discuss its behavior as a function of the length of the series, and explain how other quantities of interest studied previously in learning theory - as well as in dynamical systems and statistical mechanics - emerge from this universally definable concept. We then prove that predictive information provides the {\em unique measure for the complexity} of dynamics underlying the time series and show that there are classes of models characterized by {\em power-law growth of the predictive information} that are qualitatively more complex than any of the systems that have been investigated before. Further, we investigate numerically the learning of a nonparametric probability density, which is an example of a problem with power-law complexity, and show that the proper Bayesian formulation of this problem provides for the `Occam' factors that punish overly complex models and thus allow one {\em to learn not only a solution within a specific model class, but also the class itself} using the data only and with very few a priori assumptions. We study a possible {\em information theoretic method} that regularizes the learning of an undersampled discrete variable, and show that learning in such a setup goes through stages of very different complexities. Finally, we discuss how all of these ideas may be useful in various problems in physics, statistics, and, most importantly, biology.

140 pages; Ph.D. thesis, Princeton University, department of Physics

References in corpus (12)

• The information bottleneck method
• Computational Mechanics: Pattern and Prediction, Structure and Simplicity
• Entropy and Long range correlations in literary English
• Entropy and Long range correlations in literary English
• Field Theories for Learning Probability Distributions
• Information Bottlenecks, Causal States, and Statistical Relevance Bases: How to Represent Relevant Information in Memoryless Transduction
• The Analysis of Data from Continuous Probability Distributions
• Predictive Information
• Reparametrization invariant statistical inference and gravity
• Field Theoretical Analysis of On-line Learning of Probability Distributions
• Statistical measures of complexity for strongly interacting systems
• Predictability, complexity and learning

Cited by in corpus (7)

• Entropy and information in neural spike trains: Progress on the sampling problem
• The Organization of Intrinsic Computation: Complexity-Entropy Diagrams and the Diversity of Natural Information Processing
• Structural Information in Two-Dimensional Patterns: Entropy Convergence and Excess Entropy
• Differential Entropy Rate Characterisations of Long Range Dependent Processes
• Predictability, complexity and learning
• Invertible generalized synchronization: A putative mechanism for implicit learning in biological and artificial neural systems
• Fluctuation-dissipation theorem and models of learning