Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Abstract

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.

The first two authors contributed equally to this work

References in corpus (5)

• Unsupervised Learning for Physical Interaction through Video Prediction
• Early Visual Concept Learning with Unsupervised Deep Learning
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
• An Uncertain Future: Forecasting from Static Images using Variational Autoencoders

Cited by in corpus (100)

• Toward Multimodal Image-to-Image Translation
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• A Review on Deep Learning Techniques for Video Prediction
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Unsupervised Learning for Physical Interaction through Video Prediction
• Stochastic Adversarial Video Prediction
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Video-to-Video Synthesis
• Self-supervised learning of a facial attribute embedding from video
• Learning to Decompose and Disentangle Representations for Video Prediction
• Unsupervised Learning of Object Keypoints for Perception and Control
• Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
• Photographic Image Synthesis with Cascaded Refinement Networks
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Stochastic Variational Video Prediction
• Transformation-Based Models of Video Sequences
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Learning Beyond Human Expertise with Generative Models for Dental Restorations
• DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
• Unsupervised learning of foreground object detection
• Tackling Over-pruning in Variational Autoencoders
• Predictive-Corrective Networks for Action Detection
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Predicting Deeper into the Future of Semantic Segmentation
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Burst Denoising with Kernel Prediction Networks
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction
• Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective
• Video Imagination from a Single Image with Transformation Generation
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
• Video Compression through Image Interpolation
• Unsupervised Learning of Object Structure and Dynamics from Videos
• Dual Motion GAN for Future-Flow Embedded Video Prediction
• Hybrid Learning of Optical Flow and Next Frame Prediction to Boost Optical Flow in the Wild
• Novel Video Prediction for Large-scale Scene using Optical Flow
• Motion Prediction Under Multimodality with Conditional Stochastic Networks
• Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Improving Video Generation for Multi-functional Applications
• Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing
• Ego-Pose Estimation and Forecasting as Real-Time PD Control
• Im2Flow: Motion Hallucination from Static Images for Action Recognition
• Flow-Grounded Spatial-Temporal Video Prediction from Still Images
• Learning 3D Human Dynamics from Video
• Deep Online Video Stabilization
• Self-Supervised Relative Depth Learning for Urban Scene Understanding
• Few-shot Video-to-Video Synthesis
• Learning Diverse Image Colorization
• Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss
• Learning to Represent Mechanics via Long-term Extrapolation and Interpolation
• Super-Resolution with Deep Adaptive Image Resampling
• Basis Prediction Networks for Effective Burst Denoising with Large Kernels
• Relational Action Forecasting
• Interpretable Deep Feature Propagation for Early Action Recognition
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Video Interpolation and Prediction with Unsupervised Landmarks
• Learning to Take Directions One Step at a Time
• Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction
• Precipitation nowcasting using a stochastic variational frame predictor with learned prior distribution
• Meta-Learning Dynamics Forecasting Using Task Inference
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
• Flow-based Intrinsic Curiosity Module
• Probabilistic Video Generation using Holistic Attribute Control
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• Disentangling Motion, Foreground and Background Features in Videos
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
• Dynamic Filtering with Large Sampling Field for ConvNets
• Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos
• Learning Temporal Dynamics from Cycles in Narrated Video
• Learning to Forecast Videos of Human Activity with Multi-granularity Models and Adaptive Rendering
• Neural Allocentric Intuitive Physics Prediction from Real Videos
• Learning Disentangled Representations of Video with Missing Data
• Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
• Video Time: Properties, Encoders and Evaluation
• Painting Many Pasts: Synthesizing Time Lapse Videos of Paintings
• Recurrent Flow-Guided Semantic Forecasting
• Visual Data Augmentation through Learning
• Convolutional Humanoid Animation via Deformation
• PHYRE: A New Benchmark for Physical Reasoning
• Forecasting Hands and Objects in Future Frames
• Interpretable Intuitive Physics Model
• Tool as Embodiment for Recursive Manipulation
• Compositional Video Prediction
• Dynamic Variational Autoencoders for Visual Process Modeling
• Do End-to-end Stereo Algorithms Under-utilize Information?
• InterpolationSLAM: A Novel Robust Visual SLAM System in Rotating Scenes
• Self-Supervised Equivariant Scene Synthesis from Video
• Authoring image decompositions with generative models
• Point-to-Point Video Generation
• VP-GO: a "light" action-conditioned visual prediction model
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• Visual Reaction: Learning to Play Catch with Your Drone
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning