Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

Abstract

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.

arXiv commentJournal preprint of arXiv:1607.02586 (IEEE TPAMI, 2019). The first two authors contributed equally to this work. Project page: http://visualdynamics.csail.mit.edu

References in corpus (11)

• Auto-Encoding Variational Bayes
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• Semi-Supervised Learning with Deep Generative Models
• Tutorial on Variational Autoencoders
• Generating Videos with Scene Dynamics
• Dynamic Filter Networks
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Early Visual Concept Learning with Unsupervised Deep Learning
• Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet

Cited by in corpus (5)

• Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions
• Hybrid physics-AI outperforms numerical weather prediction for extreme precipitation nowcasting
• Extracting Interpretable Physical Parameters from Spatiotemporal Systems using Unsupervised Learning
• Controllable and Progressive Image Extrapolation
• Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis