Hierarchical Long-term Video Prediction without Supervision

Abstract

Much of recent research has been devoted to video prediction and generation, yet most of the previous works have demonstrated only limited success in generating videos on short-term horizons. The hierarchical video prediction method by Villegas et al. (2017) is an example of a state-of-the-art method for long-term video prediction, but their method is limited because it requires ground truth annotation of high-level structures (e.g., human joint landmarks) at training time. Our network encodes the input frame, predicts a high-level encoding into the future, and then a decoder with access to the first frame produces the predicted image from the predicted encoding. The decoder also produces a mask that outlines the predicted foreground object (e.g., person) as a by-product. Unlike Villegas et al. (2017), we develop a novel training method that jointly trains the encoder, the predictor, and the decoder together without highlevel supervision; we further improve upon this by using an adversarial loss in the feature space to train the predictor. Our method can predict about 20 seconds into the future and provides better results compared to Denton and Fergus (2018) and Finn et al. (2016) on the Human 3.6M dataset.

arXiv commentInternational Conference on Machine Learning (ICML) 2018

Cited by in corpus (20)

• A Review on Deep Learning Techniques for Video Prediction
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
• How do Mixture Density RNNs Predict the Future?
• Unsupervised Learning of Object Structure and Dynamics from Videos
• GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis
• Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
• Disentangling Propagation and Generation for Video Prediction
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Video Interpolation and Prediction with Unsupervised Landmarks
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Deep Learned Frame Prediction for Video Compression
• Motion Selective Prediction for Video Frame Synthesis
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• Variational Predictive Routing with Nested Subjective Timescales
• Stochastic Dynamics for Video Infilling
• Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos