Compositional Video Prediction

Abstract

We present an approach for pixel-level future prediction given an input image of a scene. We observe that a scene is comprised of distinct entities that undergo motion and present an approach that operationalizes this insight. We implicitly predict future states of independent entities while reasoning about their interactions, and compose future video frames using these predicted states. We overcome the inherent multi-modality of the task using a global trajectory-level latent random variable, and show that this allows us to sample diverse and plausible futures. We empirically validate our approach against alternate representations and ways of incorporating multi-modality. We examine two datasets, one comprising of stacked objects that may fall, and the other containing videos of humans performing activities in a gym, and show that our approach allows realistic stochastic video prediction across these diverse settings. See https://judyye.github.io/CVP/ for video predictions.

arXiv commentaccepted to ICCV19

References in corpus (12)

• Auto-Encoding Variational Bayes
• Interaction Networks for Learning about Objects, Relations and Physics
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Stochastic Video Generation with a Learned Prior
• Learning to Generate Long-term Future via Hierarchical Prediction
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Learning Physical Intuition of Block Towers by Example
• Visual Interaction Networks
• To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
• Video Imagination from a Single Image with Transformation Generation
• Motion Prediction Under Multimodality with Conditional Stochastic Networks