Stochastic Latent Residual Video Prediction

Abstract

Designing video prediction models that account for the inherent uncertainty of the future is challenging. Most works in the literature are based on stochastic image-autoregressive recurrent networks, which raises several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and temporal dynamics. However, no such model for stochastic video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model whose dynamics are governed in a latent space by a residual update rule. This first-order scheme is motivated by discretization schemes of differential equations. It naturally models video dynamics as it allows our simpler, more interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

Cited by in corpus (13)

• CCVS: Context-aware Controllable Video Synthesis
• FitVid: Overfitting in Pixel-Level Video Prediction
• Clockwork Variational Autoencoders
• PDE-Driven Spatiotemporal Disentanglement
• MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Deep Video Prediction for Time Series Forecasting
• Wide and Narrow: Video Prediction from Context and Motion
• Latent Neural Differential Equations for Video Generation
• Structural Forecasting for Tropical Cyclone Intensity Prediction: Providing Insight with Deep Learning
• Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure
• Layered Controllable Video Generation
• Taylor saves for later: disentanglement for video prediction using Taylor representation