Video Ladder Networks

Abstract

We present the Video Ladder Network (VLN) for efficiently generating future video frames. VLN is a neural encoder-decoder model augmented at all layers by both recurrent and feedforward lateral connections. At each layer, these connections form a lateral recurrent residual block, where the feedforward connection represents a skip connection and the recurrent connection represents the residual. Thanks to the recurrent connections, the decoder can exploit temporal summaries generated from all layers of the encoder. This way, the top layer is relieved from the pressure of modeling lower-level spatial and temporal details. Furthermore, we extend the basic version of VLN to incorporate ResNet-style residual blocks in the encoder and decoder, which help improving the prediction results. VLN is trained in self-supervised regime on the Moving MNIST dataset, achieving competitive results while having very simple structure and providing fast inference.

This version extends the paper accepted at the NIPS 2016 workshop on ML for Spatiotemporal Forecasting, with more details and more experimental results

References in corpus (1)

• Spatio-temporal video autoencoder with differentiable memory

Cited by in corpus (10)

• A Review on Deep Learning Techniques for Video Prediction
• Symbolic Pregression: Discovering Physical Laws from Distorted Video
• Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
• Frequency Domain Transformer Networks for Video Prediction
• Motion Segmentation using Frequency Domain Transformer Networks
• Deep Learned Frame Prediction for Video Compression
• End-to-End Learning for Video Frame Compression with Self-Attention
• Recurrent Flow-Guided Semantic Forecasting
• Sampling-free Uncertainty Estimation in Gated Recurrent Units with Exponential Families
• Predicting the Future with Transformational States