Disentangling Propagation and Generation for Video Prediction

Abstract

A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated. Prior approaches to video prediction typically learn either to warp or to hallucinate future pixels, but not both. In this paper, we describe a computational model for high-fidelity video prediction which disentangles motion-specific propagation from motion-agnostic generation. We introduce a confidence-aware warping operator which gates the output of pixel predictions from a flow predictor for non-occluded regions and from a context encoder for occluded regions. Moreover, in contrast to prior works where confidence is jointly learned with flow and appearance using a single network, we compute confidence after a warping step, and employ a separate network to inpaint exposed regions. Empirical results on both synthetic and real datasets show that our disentangling approach provides better occlusion maps and produces both sharper and more realistic predictions compared to strong baselines.

arXiv commentICCV 2019

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Generative Adversarial Text to Image Synthesis
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Generating Videos with Scene Dynamics
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Stochastic Video Generation with a Learned Prior
• High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
• Video (language) modeling: a baseline for generative models of natural videos
• Stochastic Adversarial Video Prediction
• Learning to Generate Long-term Future via Hierarchical Prediction
• Hierarchical Long-term Video Prediction without Supervision
• Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
• Deep Layer Aggregation
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Dense Pose Transfer
• ContextVP: Fully Context-Aware Video Prediction