Explicit mutual information for simple networks and neurons with lognormal activities

Abstract

Networks with stochastic variables described by heavy tailed lognormal distribution are ubiquitous in nature, and hence they deserve an exact information-theoretic characterization. We derive analytical formulas for mutual information between elements of different networks with correlated lognormally distributed activities. In a special case, we find an explicit expression for mutual information between neurons when neural activities and synaptic weights are lognormally distributed, as suggested by experimental data. Comparison of this expression with the case when these two variables have short tails, reveals that mutual information with heavy tails for neurons and synapses is generally larger and can diverge for some finite variances in presynaptic firing rates and synaptic weights. This result suggests that evolution might prefer brains with heterogeneous dynamics to optimize information processing.

Corrected version: corrected and expanded discussion of neural mutual information (lognormal vs. gaussian distributions)

References in corpus (12)

• Statistical mechanics of complex networks
• The structure and function of complex networks
• Stochastic thermodynamics, fluctuation theorems, and molecular machines
• The Paradox of the expected Time until the Next Earthquake
• Entropy and mutual information in models of deep neural networks
• Quantum Mutual Information as a Probe for Many-Body Localization
• Determinants of Block Tridiagonal Matrices
• Broad distribution effects in sums of lognormal random variables
• Correlations in Bipartite Collaboration Networks
• On the Mutual Information in Hawking Radiation
• Truncated lognormal distributions and scaling in the size of naturally defined population clusters
• Probing high order dependencies with information theory