A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions

Abstract

Despite the impressive performance of biological and artificial networks, an intuitive understanding of how their local learning dynamics contribute to network-level task solutions remains a challenge to this date. Efforts to bring learning to a more local scale indeed lead to valuable insights, however, a general constructive approach to describe local learning goals that is both interpretable and adaptable across diverse tasks is still missing. We have previously formulated a local information processing goal that is highly adaptable and interpretable for a model neuron with compartmental structure. Building on recent advances in Partial Information Decomposition (PID), we here derive a corresponding parametric local learning rule, which allows us to introduce 'infomorphic' neural networks. We demonstrate the versatility of these networks to perform tasks from supervised, unsupervised and memory learning. By leveraging the interpretable nature of the PID framework, infomorphic networks represent a valuable tool to advance our understanding of the intricate structure of local learning.

30 pages, 14 figures

References in corpus (19)

• Quantifying unique information
• A Bivariate Measure of Redundant Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Measuring multivariate redundant information with pointwise common change in surprisal
• Bits from Biology for Computational Intelligence
• Dendritic predictive coding: A theory of cortical computation with spiking neurons
• Training Neural Networks with Local Error Signals
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Dendritic cortical microcircuits approximate the backpropagation algorithm
• Introducing a differentiable measure of pointwise shared information
• Estimating Information Flow in Deep Neural Networks
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• A Measure of the Complexity of Neural Representations based on Partial Information Decomposition
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• What should a neuron aim for? Designing local objective functions based on information theory
• Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons

Cited by in corpus (1)

• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions