Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Abstract

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.

The first two authors contributed equally to this work

References in corpus (1)

• An Uncertain Future: Forecasting from Static Images using Variational Autoencoders

Cited by in corpus (16)

• Photographic Image Synthesis with Cascaded Refinement Networks
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Tackling Over-pruning in Variational Autoencoders
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• Video Imagination from a Single Image with Transformation Generation
• Dual Motion GAN for Future-Flow Embedded Video Prediction
• Super-Resolution with Deep Adaptive Image Resampling
• Learning to Represent Mechanics via Long-term Extrapolation and Interpolation
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Precipitation nowcasting using a stochastic variational frame predictor with learned prior distribution
• Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction
• Disentangling Motion, Foreground and Background Features in Videos
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
• Learning to Forecast Videos of Human Activity with Multi-granularity Models and Adaptive Rendering
• Visual Data Augmentation through Learning
• Authoring image decompositions with generative models