Energy--Information Trade-off Induces Continuous and Discontinuous Phase Transitions in Lateral Predictive Coding

Abstract

Lateral predictive coding is a recurrent neural network which creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here we investigate the trade-off between information robustness and energy in a linear model of lateral predictive coding analytically and by numerical minimization of a free energy. We observe several phase transitions in the synaptic weight matrix, especially a continuous transition which breaks reciprocity and permutation symmetry and builds cyclic dominance and a discontinuous transition with the associated sudden emergence of tight balance between excitatory and inhibitory interactions. The optimal network follows an ideal-gas law in an extended temperature range and saturates the efficiency upper-bound of energy utilization. These results bring theoretical insights on the emergence and evolution of complex internal models in predictive processing systems.

arXiv comment6 pages main text, supplementary text combined. This is an extensively revised version, containing new analytical results and numerical results

References in corpus (12)

• Weak pairwise correlations imply strongly correlated network states in a neural population
• Recurrent computations for visual pattern completion
• Random numbers for large scale distributed Monte Carlo simulations
• Less is More: Wiring-Economical Modular Networks Support Self-Sustained Firing-Economical Neural Avalanches for Efficient Processing
• The Optimal Size of Stochastic Hodgkin-Huxley Neuronal Systems for Maximal Energy Efficiency in Coding of Pulse Signals
• Inferring Global Dynamics of a Black-Box System Using Machine Learning
• Copy the dynamics using a learning machine
• Inferring Global Dynamics Using a Learning Machine
• The information content of brain states is explained by structural constraints on state energetics
• Bayesian mechanics of perceptual inference and motor control in the brain
• Moderate Point: Balanced Entropy and Enthalpy Contributions in Soft Matter
• Lateral predictive coding revisited: Internal model, symmetry breaking, and response time

Cited by in corpus (2)

• Discontinuous phase transition of feature detection in lateral predictive coding
• Network Dynamics-Based Framework for Understanding Deep Neural Networks