Deep Predictive Coding Networks

Abstract

The quality of data representation in deep learning methods is directly related to the prior model imposed on the representations; however, generally used fixed priors are not capable of adjusting to the context in the data. To address this issue, we propose deep predictive coding networks, a hierarchical generative model that empirically alters priors on the latent representations in a dynamic and context-sensitive manner. This model captures the temporal dependencies in time-varying signals and uses top-down information to modulate the representation in lower layers. The centerpiece of our model is a novel procedure to infer sparse states of a dynamic model which is used for feature extraction. We also extend this feature extraction block to introduce a pooling function that captures locally invariant representations. When applied on a natural video data, we show that our method is able to learn high-level visual features. We also demonstrate the role of the top-down connections by showing the robustness of the proposed model to structured noise.

13 Pages, 7 figures, submission for ICLR 2013

References in corpus (1)

• Smoothing proximal gradient method for general structured sparse regression

Cited by in corpus (20)

• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• A Review on Deep Learning Techniques for Video Prediction
• Unsupervised Learning of Visual Structure using Predictive Generative Networks
• Deep Predictive Coding Network for Object Recognition
• PreCNet: Next-Frame Video Prediction Based on Predictive Coding
• Predictive Coding: a Theoretical and Experimental Review
• Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics
• A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception
• PredNet and Predictive Coding: A Critical Review
• Prediction Under Uncertainty with Error-Encoding Networks
• CortexNet: a Generic Network Family for Robust Visual Temporal Representations
• Learning to Adapt by Minimizing Discrepancy
• Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All Networks
• Predictive coding feedback results in perceived illusory contours in a recurrent neural network
• CTNN: Corticothalamic-inspired neural network
• Predictive Coding Networks and Inference Learning: Tutorial and Survey
• Dynamic Variational Autoencoders for Visual Process Modeling