An Information-Theoretic Formalism for Multiscale Structure in Complex Systems

Abstract

We develop a general formalism for representing and understanding structure in complex systems. In our view, structure is the totality of relationships among a system's components, and these relationships can be quantified using information theory. In the interest of flexibility we allow information to be quantified using any function, including Shannon entropy and Kolmogorov complexity, that satisfies certain fundamental axioms. Using these axioms, we formalize the notion of a dependency among components, and show how a system's structure is revealed in the amount of information assigned to each dependency. We explore quantitative indices that summarize system structure, providing a new formal basis for the complexity profile and introducing a new index, the "marginal utility of information". Using simple examples, we show how these indices capture intuitive ideas about structure in a quantitative way. Our formalism also sheds light on a longstanding mystery: that the mutual information of three or more variables can be negative. We discuss applications to complex networks, gene regulation, the kinetic theory of fluids and multiscale cybernetic thermodynamics.

60 pages, 10 figures

References in corpus (30)

• Emergence of scaling in random networks
• The structure and function of complex networks
• Statistical physics of social dynamics
• Traffic and Related Self-Driven Many-Particle Systems
• The information bottleneck method
• Hierarchical Structure in Financial Markets
• Weak pairwise correlations imply strongly correlated network states in a neural population
• Effect of Trends on Detrended Fluctuation Analysis
• Information heat engine: converting information to energy by feedback control
• Epidemic spreading in correlated complex networks
• High-precision test of Landauer's principle in a feedback trap
• The thermodynamic meaning of negative entropy
• Experimental realization of a Szilard engine with a single electron
• Network information and connected correlations
• Nonequilibrium Thermodynamics of Feedback Control
• Entropy Optimization of Scale-Free Networks Robustness to Random Failures
• Shannon Information and Kolmogorov Complexity
• Entropic Bell Inequalities
• Time's Barbed Arrow: Irreversibility, Crypticity, and Stored Information
• An Information-Theoretic Approach to Network Modularity
• Prediction, Retrodiction, and The Amount of Information Stored in the Present
• Clustering of correlated networks
• Self-assembly, modularity and physical complexity
• Quantifying and Visualizing Attribute Interactions
• Information Bottlenecks, Causal States, and Statistical Relevance Bases: How to Represent Relevant Information in Memoryless Transduction
• Causal Markov condition for submodular information measures
• Multiscale Complexity of Correlated Gaussians
• Eco-Evolutionary Feedback in Host--Pathogen Spatial Dynamics
• The Category-Theoretic Arithmetic of Information
• Principles of Security: Human, Cyber, and Biological

Cited by in corpus (4)

• Multivariate Dependence Beyond Shannon Information
• Hidden Structures of Information Transport Underlying Spiral Wave Dynamics
• Stripping syntax from complexity: An information-theoretical perspective on complex systems
• Functional Dynamical Structures in Complex Systems: an Information-Theoretic Approach