On sampling and modeling complex systems

Abstract

The study of complex systems is limited by the fact that only few variables are accessible for modeling and sampling, which are not necessarily the most relevant ones to explain the systems behavior. In addition, empirical data typically under sample the space of possible states. We study a generic framework where a complex system is seen as a system of many interacting degrees of freedom, which are known only in part, that optimize a given function. We show that the underlying distribution with respect to the known variables has the Boltzmann form, with a temperature that depends on the number of unknown variables. In particular, when the unknown part of the objective function decays faster than exponential, the temperature decreases as the number of variables increases. We show in the representative case of the Gaussian distribution, that models are predictable only when the number of relevant variables is less than a critical threshold. As a further consequence, we show that the information that a sample contains on the behavior of the system is quantified by the entropy of the frequency with which different states occur. This allows us to characterize the properties of maximally informative samples: in the under-sampling regime, the most informative frequency size distributions have power law behavior and Zipf's law emerges at the crossover between the under sampled regime and the regime where the sample contains enough statistics to make inference on the behavior of the system. These ideas are illustrated in some applications, showing that they can be used to identify relevant variables or to select most informative representations of data, e.g. in data clustering.

28 pages, 7 figures, version accepted for publication

References in corpus (1)

• Power-law distributions in empirical data

Cited by in corpus (26)

• Colloquium: Criticality and dynamical scaling in living systems
• Inverse statistical problems: from the inverse Ising problem to data science
• Zipf's law and criticality in multivariate data without fine-tuning
• Statistical laws in linguistics
• Signatures of criticality arise in simple neural population models with correlations
• Statistical Criticality arises in Most Informative Representations
• Predictability of extreme events in social media
• Criticality of mostly informative samples: A Bayesian model selection approach
• Quantifying Relevance in Learning and Inference
• Identifying relevant positions in proteins by Critical Variable Selection
• Maximal Diversity and Zipf's Law
• Ensemble Inhibition and Excitation in the Human Cortex: an Ising Model Analysis with Uncertainties
• A statistical physics perspective on criticality in financial markets
• Making sense of complex systems through resolution, relevance, and mapping entropy
• Cooperation, competition and the emergence of criticality in communities of adaptive systems
• Dynamical signatures of structural connectivity damage to a model of the brain posed at criticality
• Large Deviations Properties of Maximum Entropy Markov Chains from Spike Trains
• A random energy approach to deep learning
• Entropic determination of the phase transition in a coevolving opinion-formation model
• Extrinsic vs Intrinsic Criticality in Systems with Many Components
• Scale-invariant representation of machine learning
• Perceptual Rationality: An Evolutionary Game Theory of Perceptually Rational Decision-Making
• The peculiar statistical mechanics of Optimal Learning Machines
• The dual of the space of interactions in neural network models
• Filtering Statistics on Networks
• Complex distributions emerging in filtering and compression