PreCNet: Next-Frame Video Prediction Based on Predictive Coding

Abstract

Predictive coding, currently a highly influential theory in neuroscience, has not been widely adopted in machine learning yet. In this work, we transform the seminal model of Rao and Ballard (1999) into a modern deep learning framework while remaining maximally faithful to the original schema. The resulting network we propose (PreCNet) is tested on a widely used next frame video prediction benchmark, which consists of images from an urban environment recorded from a car-mounted camera, and achieves state-of-the-art performance. Performance on all measures (MSE, PSNR, SSIM) was further improved when a larger training set (2M images from BDD100k), pointing to the limitations of the KITTI training set. This work demonstrates that an architecture carefully based in a neuroscience model, without being explicitly tailored to the task at hand, can exhibit exceptional performance.

arXiv commentAccepted by IEEE Transactions on Neural Networks and Learning Systems (TNNLS)

References in corpus (9)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• LSTM: A Search Space Odyssey
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• Deep Predictive Coding Network for Object Recognition
• Deep Predictive Coding Networks
• MuPNet: Multi-modal Predictive Coding Network for Place Recognition by Unsupervised Learning of Joint Visuo-Tactile Latent Representations