Information Anatomy of Stochastic Equilibria

Abstract

A stochastic nonlinear dynamical system generates information, as measured by its entropy rate. Some---the ephemeral information---is dissipated and some---the bound information---is actively stored and so affects future behavior. We derive analytic expressions for the ephemeral and bound informations in the limit of small-time discretization for two classical systems that exhibit dynamical equilibria: first-order Langevin equations (i) where the drift is the gradient of a potential function and the diffusion matrix is invertible and (ii) with a linear drift term (Ornstein-Uhlenbeck) but a noninvertible diffusion matrix. In both cases, the bound information is sensitive only to the drift, while the ephemeral information is sensitive only to the diffusion matrix and not to the drift. Notably, this information anatomy changes discontinuously as any of the diffusion coefficients vanishes, indicating that it is very sensitive to the noise structure. We then calculate the information anatomy of the stochastic cusp catastrophe and of particles diffusing in a heat bath in the overdamped limit, both examples of stochastic gradient descent on a potential landscape. Finally, we use our methods to calculate and compare approximations for the so-called time-local predictive information for adaptive agents.

arXiv comment35 pages, 3 figures, 1 table; http://csc.ucdavis.edu/~cmg/compmech/pubs/iase.htm

References in corpus (15)

• Computational Mechanics: Pattern and Prediction, Structure and Simplicity
• Thermodynamic formalism for systems with Markov dynamics
• Time's Barbed Arrow: Irreversibility, Crypticity, and Stored Information
• Statistical Complexity of Simple 1D Spin Systems
• Anatomy of a Bit: Information in a Time Series Observation
• Structural Information in Two-Dimensional Patterns: Entropy Convergence and Excess Entropy
• Prediction, Retrodiction, and The Amount of Information Stored in the Present
• Stability and Diversity in Collective Adaptation
• Information driven self-organization of complex robotic behaviors
• Bayesian Structural Inference for Hidden Processes
• Discovering Planar Disorder in Close-Packed Structures from X-Ray Diffraction: Beyond the Fault Model
• Chaos Forgets and Remembers: Measuring Information Creation, Destruction, and Storage
• Synchronization and Control in Intrinsic and Designed Computation: An Information-Theoretic Analysis of Competing Models of Stochastic Computation
• Predictive information in a nonequilibrium critical model
• Predictive information in a sensory population

Cited by in corpus (11)

• Stochastic thermodynamics of Langevin systems under time-delayed feedback control: I. Second-law-like inequalities
• Editorial Comment on the Special Issue of "Information in Dynamical Systems and Complex Systems"
• Informational and Causal Architecture of Discrete-Time Renewal Processes
• Informational and Causal Architecture of Continuous-time Renewal and Hidden Semi-Markov Processes
• Superior memory efficiency of quantum devices for the simulation of continuous-time stochastic processes
• Shannon Entropy Rate of Hidden Markov Processes
• Characterising information-theoretic storage and transfer in continuous time processes
• Statistical Signatures of Structural Organization: The case of long memory in renewal processes
• Local Causal States and Discrete Coherent Structures
• Circumventing the Curse of Dimensionality in Prediction: Causal Rate-Distortion for Infinite-Order Markov Processes
• The Origins of Computational Mechanics: A Brief Intellectual History and Several Clarifications