Anatomy of a Bit: Information in a Time Series Observation

Abstract

Appealing to several multivariate information measures---some familiar, some new here---we analyze the information embedded in discrete-valued stochastic time series. We dissect the uncertainty of a single observation to demonstrate how the measures' asymptotic behavior sheds structural and semantic light on the generating process's internal information dynamics. The measures scale with the length of time window, which captures both intensive (rates of growth) and subextensive components. We provide interpretations for the components, developing explicit relationships between them. We also identify the informational component shared between the past and the future that is not contained in a single observation. The existence of this component directly motivates the notion of a process's effective (internal) states and indicates why one must build models.

15 pages, 12 figures, 2 tables; http://cse.ucdavis.edu/~cmg/compmech/pubs/anatomy.htm

References in corpus (4)

• Nonnegative Decomposition of Multivariate Information
• Identification of functional information subgraphs in complex networks
• Synchronization and Control in Intrinsic and Designed Computation: An Information-Theoretic Analysis of Competing Models of Stochastic Computation
• A measure of statistical complexity based on predictive information

Cited by in corpus (32)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Bits from Biology for Computational Intelligence
• Multivariate Dependence Beyond Shannon Information
• Integrated information as a common signature of dynamical and information-processing complexity
• Range entropy: A bridge between signal complexity and self-similarity
• Understanding interdependency through complex information sharing
• What did Erwin Mean? The Physics of Information from the Materials Genomics of Aperiodic Crystals and Water to Molecular Information Catalysts and Life
• Chaos Forgets and Remembers: Measuring Information Creation, Destruction, and Storage
• Informational and Causal Architecture of Continuous-time Renewal and Hidden Semi-Markov Processes
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Many Roads to Synchrony: Natural Time Scales and Their Algorithms
• Informational and Causal Architecture of Discrete-Time Renewal Processes
• Superior memory efficiency of quantum devices for the simulation of continuous-time stochastic processes
• Information Symmetries in Irreversible Processes
• Information Anatomy of Stochastic Equilibria
• How Hidden are Hidden Processes? A Primer on Crypticity and Entropy Convergence
• From Microscopic Heterogeneity to Macroscopic Complexity in the Contrarian Voter Model
• Optimized Bacteria are Environmental Prediction Engines
• Characterising information-theoretic storage and transfer in continuous time processes
• Flickering emergences: The question of locality in information-theoretic approaches to emergence
• Spectral Simplicity of Apparent Complexity, Part I: The Nondiagonalizable Metadynamics of Prediction
• Estimating Functions of Distributions Defined over Spaces of Unknown Size
• The Elusive Present: Hidden Past and Future Dependency and Why We Build Models
• Sequential Complexity as a Descriptor for Musical Similarity
• Measurement-Induced Randomness and Structure in Controlled Qubit Processes
• Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle
• Ambiguity Rate of Hidden Markov Processes
• Trimming the Independent Fat: Sufficient Statistics, Mutual Information, and Predictability from Effective Channel States
• Exo-Daisy World: Revisiting Gaia Theory through an Informational Architecture Perspective
• Surveying the space of descriptions of a composite system with machine learning
• The fundamental thermodynamic bounds on finite models
• Intrinsic computation of a Monod-Wyman-Changeux molecule