Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions

Abstract

In many neural systems anatomical motifs are present repeatedly, but despite their structural similarity they can serve very different tasks. A prime example for such a motif is the canonical microcircuit of six-layered neo-cortex, which is repeated across cortical areas, and is involved in a number of different tasks (e.g.sensory, cognitive, or motor tasks). This observation has spawned interest in finding a common underlying principle, a 'goal function', of information processing implemented in this structure. By definition such a goal function, if universal, cannot be cast in processing-domain specific language (e.g. 'edge filtering', 'working memory'). Thus, to formulate such a principle, we have to use a domain-independent framework. Information theory offers such a framework. However, while the classical framework of information theory focuses on the relation between one input and one output (Shannon's mutual information), we argue that neural information processing crucially depends on the combination of \textit{multiple} inputs to create the output of a processor. To account for this, we use a very recent extension of Shannon Information theory, called partial information decomposition (PID). PID allows to quantify the information that several inputs provide individually (unique information), redundantly (shared information) or only jointly (synergistic information) about the output. First, we review the framework of PID. Then we apply it to reevaluate and analyze several earlier proposals of information theoretic neural goal functions (predictive coding, infomax, coherent infomax, efficient coding). We find that PID allows to compare these goal functions in a common framework, and also provides a versatile approach to design new goal functions from first principles. Building on this, we design and analyze a novel goal function, called 'coding with synergy'. [...]

21 pages, 4 figures, appendix

References in corpus (4)

• Local information transfer as a spatiotemporal filter for complex systems
• Nonnegative Decomposition of Multivariate Information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Generalized Measures of Information Transfer

Cited by in corpus (26)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Measuring multivariate redundant information with pointwise common change in surprisal
• Control of criticality and computation in spiking neuromorphic networks with plasticity
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Introducing a differentiable measure of pointwise shared information
• Dynamical independence: discovering emergent macroscopic processes in complex dynamical systems
• A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells
• Generalised Measures of Multivariate Information Content
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• On extractable shared information
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• A generalization of the maximum entropy principle for curved statistical manifolds
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Interplay of synergy and redundancy in diamond motif
• Pooling Probability Distributions and the Partial Information Decomposition
• Higher-Order Interactions and Their Duals Reveal Synergy and Logical Dependence beyond Shannon-Information
• Computationally Efficient Approximations for Matrix-based Renyi's Entropy
• Contrasting information theoretic decompositions of modulatory and arithmetic interactions in neural information processing systems
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Quantifying High-Order Interdependencies in Entangled Quantum States
• Partial information decomposition: redundancy as information bottleneck
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective