Data-driven inference of hidden nodes in networks

Abstract

The explosion of activity in finding interactions in complex systems is driven by availability of copious observations of complex natural systems. However, such systems, e.g. the human brain, are rarely completely observable. Interaction network inference must then contend with hidden variables affecting the behavior of the observed parts of the system. We present a novel data-driven approach for model inference with hidden variables. From configurations of observed variables, we identify the observed-to-observed, hidden-to-observed, observed-to-hidden, and hidden-to-hidden interactions, the configurations of hidden variables, and the number of hidden variables. We demonstrate the performance of our method by simulating a kinetic Ising model, and show that our method outperforms existing methods. Turning to real data, we infer the hidden nodes in a neuronal network in the salamander retina and a stock market network. We show that predictive modeling with hidden variables is significantly more accurate than that without hidden variables. Finally, an important hidden variable problem is to find the number of clusters in a dataset. We apply our method to classify MNIST handwritten digits. We find that there are about 60 clusters which are roughly equally distributed amongst the digits.

9 pages, 7 figures

References in corpus (11)

• Weak pairwise correlations imply strongly correlated network states in a neural population
• Whole-brain calcium imaging with cellular resolution in freely behaving C. elegans
• Searching for collective behavior in a network of real neurons
• Mean Field Theory For Non-Equilibrium Network Reconstruction
• Maximum likelihood reconstruction for Ising models with asynchronous updates
• Learning and inference in a nonequilibrium Ising model with hidden nodes
• Market structure explained by pairwise interactions
• Network Inference with Hidden Units
• Belief-Propagation and replicas for inference and learning in a kinetic Ising model with hidden spins
• Inferring hidden states in a random kinetic Ising model: replica analysis
• Causality inference in stochastic systems from neurons to currencies: Profiting from small sample size

Cited by in corpus (5)

• Tackling the subsampling problem to infer collective properties from limited data
• Methods for forecasting the effect of exogenous risk on stock markets
• Inference of stochastic time series with missing data
• Reconstructing Network Structures from Partial Measurements
• Inverse Ising inference from high-temperature re-weighting of observations