Video Interpolation and Prediction with Unsupervised Landmarks

Abstract

Prediction and interpolation for long-range video data involves the complex task of modeling motion trajectories for each visible object, occlusions and dis-occlusions, as well as appearance changes due to viewpoint and lighting. Optical flow based techniques generalize but are suitable only for short temporal ranges. Many methods opt to project the video frames to a low dimensional latent space, achieving long-range predictions. However, these latent representations are often non-interpretable, and therefore difficult to manipulate. This work poses video prediction and interpolation as unsupervised latent structure inference followed by a temporal prediction in this latent space. The latent representations capture foreground semantics without explicit supervision such as keypoints or poses. Further, as each landmark can be mapped to a coordinate indicating where a semantic part is positioned, we can reliably interpolate within the coordinate domain to achieve predictable motion interpolation. Given an image decoder capable of mapping these landmarks back to the image domain, we are able to achieve high-quality long-range video interpolation and extrapolation by operating on the landmark representation space.

arXiv commentTechnical Report

References in corpus (11)

• Spectral Normalization for Generative Adversarial Networks
• Density estimation using Real NVP
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Hierarchical Long-term Video Prediction without Supervision
• Stochastic Variational Video Prediction
• Video Extrapolation with an Invertible Linear Embedding