Predictive Coding Networks and Inference Learning: Tutorial and Survey

Abstract

Recent years have witnessed a growing call for renewed emphasis on neuroscience-inspired approaches in artificial intelligence research, under the banner of NeuroAI. A prime example of this is predictive coding networks (PCNs), based on the neuroscientific framework of predictive coding. This framework views the brain as a hierarchical Bayesian inference model that minimizes prediction errors through feedback connections. Unlike traditional neural networks trained with backpropagation (BP), PCNs utilize inference learning (IL), a more biologically plausible algorithm that explains patterns of neural activity that BP cannot. Historically, IL has been more computationally intensive, but recent advancements have demonstrated that it can achieve higher efficiency than BP with sufficient parallelization. Furthermore, PCNs can be mathematically considered a superset of traditional feedforward neural networks (FNNs), significantly extending the range of trainable architectures. As inherently probabilistic (graphical) latent variable models, PCNs provide a versatile framework for both supervised learning and unsupervised (generative) modeling that goes beyond traditional artificial neural networks. This work provides a comprehensive review and detailed formal specification of PCNs, particularly situating them within the context of modern ML methods. This positions PC as a promising framework for future ML innovations.

47 pages, 11 figures, 9 tables

References in corpus (22)

• Overcoming catastrophic forgetting in neural networks
• Representation Learning with Contrastive Predictive Coding
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• A high-bias, low-variance introduction to Machine Learning for physicists
• Active Inference in Robotics and Artificial Agents: Survey and Challenges
• Deep Predictive Coding Networks
• Predictive Coding Theories of Cortical Function
• Active Predictive Coding: A Unified Neural Framework for Learning Hierarchical World Models for Perception and Planning
• On the relationship between predictive coding and backpropagation
• Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
• Brain-inspired Computational Intelligence via Predictive Coding
• Learning on Arbitrary Graph Topologies via Predictive Coding
• Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images
• A Theoretical Framework for Inference Learning
• Criticality versus uniformity in deep neural networks
• Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchy
• Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
• Curvature-Sensitive Predictive Coding with Approximate Laplace Monte Carlo
• Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
• Classification and Reconstruction Processes in Deep Predictive Coding Networks: Antagonists or Allies?
• Robust Graph Representation Learning via Predictive Coding
• JPC: Flexible Inference for Predictive Coding Networks in JAX