Tackling the subsampling problem to infer collective properties from limited data

Abstract

Complex systems are fascinating because their rich macroscopic properties emerge from the interaction of many simple parts. Understanding the building principles of these emergent phenomena in nature requires assessing natural complex systems experimentally. However, despite the development of large-scale data-acquisition techniques, experimental observations are often limited to a tiny fraction of the system. This spatial subsampling is particularly severe in neuroscience, where only a tiny fraction of millions or even billions of neurons can be individually recorded. Spatial subsampling may lead to significant systematic biases when inferring the collective properties of the entire system naively from a subsampled part. To overcome such biases, powerful mathematical tools have been developed in the past. In this perspective, we overview some issues arising from subsampling and review recently developed approaches to tackle the subsampling problem. These approaches enable one to assess, e.g., graph structures, collective dynamics of animals, neural network activity, or the spread of disease correctly from observing only a tiny fraction of the system. However, our current approaches are still far from having solved the subsampling problem in general, and hence we conclude by outlining what we believe are the main open challenges. Solving these challenges alongside the development of large-scale recording techniques will enable further fundamental insights into the working of complex and living systems.

arXiv comment20 pages, 6 figures, review article

References in corpus (67)

• Power-law distributions in empirical data
• Statistical physics of social dynamics
• Synchronization in complex networks
• Weak pairwise correlations imply strongly correlated network states in a neural population
• Scale-free correlations in bird flocks
• Homophily and Contagion Are Generically Confounded in Observational Social Network Studies
• The physics of higher-order interactions in complex systems
• Scale-free networks are rare
• Colloquium: Criticality and dynamical scaling in living systems
• Signals on Graphs: Uncertainty Principle and Sampling
• Fundamental structures of dynamic social networks
• Network information and connected correlations
• Paths to Self-Organized Criticality
• The abelian sandpile and related models
• Searching for collective behavior in a network of real neurons
• Model-free reconstruction of neuronal network connectivity from calcium imaging signals
• Inverse statistical problems: from the inverse Ising problem to data science
• Can co-location be used as a proxy for face-to-face contacts?
• Network structure from rich but noisy data
• Disorder-Induced Critical Phenomena in Hysteresis: Numerical Scaling in Three and Higher Dimensions
• Revealing networks from dynamics: an introduction
• Scale-free Networks Well Done
• Network reconstruction and community detection from dynamics
• Recent advances and open challenges in percolation
• Coarse--graining, fixed points, and scaling in a large population of neurons
• Bits from Biology for Computational Intelligence
• Mean Field Theory For Non-Equilibrium Network Reconstruction
• Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing
• Distributions of covariances as a window into the operational regime of neuronal networks
• Two types of criticality in the brain
• Inferring collective dynamical states from widely unobserved systems
• What kind of noise is brain noise: anomalous scaling behavior of the resting brain activity fluctuations
• Subsampling scaling: a theory about inference from partly observed systems
• Control of criticality and computation in spiking neuromorphic networks with plasticity
• True scale-free networks hidden by finite size effects
• Between perfectly critical and fully irregular: a reverberating model captures and predicts cortical spike propagation
• Self-organization toward criticality by synaptic plasticity
• Bayesian inference of network structure from unreliable data
• Subsampled directed-percolation models explain scaling relations experimentally observed in the brain
• Community detection in networks without observing edges
• Maximum likelihood reconstruction for Ising models with asynchronous updates
• Dynamic Adaptive Computation: Tuning network states to task requirements
• Learning and inference in a nonequilibrium Ising model with hidden nodes
• Latent dynamical variables produce signatures of spatiotemporal criticality in large biological systems
• Self-organized critical balanced networks: a unified framework
• Synaptic balance due to homeostatically self-organized quasicritical dynamics
• Causal Inference by Identification of Vector Autoregressive Processes with Hidden Components
• Testing the critical brain hypothesis using a phenomelogical renormalization group
• Avalanche Spatial Structure and Multivariable Scaling Functions; Sizes, Heights, Widths, and Views through Windows
• Criticality in spreading processes without time-scale separation and the critical brain hypothesis
• Memory functions reveal structural properties of gene regulatory networks
• Revealing biases in the sampling of ecological interaction networks
• Critical Avalanches and Subsampling in Map-based Neural Networks
• Description of spreading dynamics by microscopic network models and macroscopic branching processes can differ due to coalescence
• Unveiling causal activity of complex networks
• Inverse Ising problem in continuous time: A latent variable approach
• Quasi-stationary states in temporal correlations for traffic systems: Cologne orbital motorway as an example
• Inferring hidden states in a random kinetic Ising model: replica analysis
• MR. Estimator, a toolbox to determine intrinsic timescales from subsampled spiking activity
• A Graph Algorithmic Approach to Separate Direct from Indirect Neural Interactions
• Statistical physics approaches to subnetwork dynamics in biochemical systems
• Tractable nonlinear memory functions as a tool to capture and explain dynamical behaviours
• Data-driven inference of hidden nodes in networks
• Variational perturbation and extended Plefka approaches to dynamics on random networks: the case of the kinetic Ising model
• Characterizing spreading dynamics of subsampled systems with non-stationary external input
• Systematic model reduction captures the dynamics of extrinsic noise in biochemical subnetworks
• Detect opinion-based groups and reveal polarisation in survey data

Cited by in corpus (5)

• Synergistic signatures of group mechanisms in higher-order systems
• Nonequilibrium physics of brain dynamics
• Entropy estimators for Markovian sequences: A comparative analysis
• Model-based assessment of sampling protocols for infectious disease genomic surveillance
• From reductionism to realism: Holistic mathematical modelling for complex biological systems