High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

Abstract

Predicting future video frames is extremely challenging, as there are many factors of variation that make up the dynamics of how frames change through time. Previously proposed solutions require complex inductive biases inside network architectures with highly specialized computation, including segmentation masks, optical flow, and foreground and background separation. In this work, we question if such handcrafted architectures are necessary and instead propose a different approach: finding minimal inductive bias for video prediction while maximizing network capacity. We investigate this question by performing the first large-scale empirical study and demonstrate state-of-the-art performance by learning large models on three different datasets: one for modeling object interactions, one for modeling human motion, and one for modeling car driving.

arXiv commentIn Advances in Neural Information Processing Systems (NeurIPS), 2019

References in corpus (9)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Unsupervised Learning of Video Representations using LSTMs
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control