Stochastic Adversarial Video Prediction

Abstract

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.

arXiv commentWebsite: https://alexlee-gk.github.io/video_prediction/

References in corpus (23)

• Auto-Encoding Variational Bayes
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Unsupervised Learning of Video Representations using LSTMs
• Pixel Recurrent Neural Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Toward Multimodal Image-to-Image Translation
• Generating Videos with Scene Dynamics
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Unsupervised Learning for Physical Interaction through Video Prediction
• Unsupervised Learning of Disentangled Representations from Video
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Recurrent Environment Simulators
• Self-Supervised Visual Planning with Temporal Skip Connections
• Video Frame Interpolation via Adaptive Separable Convolution
• Transformation-Based Models of Video Sequences
• SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control
• Video Imagination from a Single Image with Transformation Generation

Cited by in corpus (98)

• A Review on Deep Learning Techniques for Video Prediction
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• VideoBERT: A Joint Model for Video and Language Representation Learning
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• Video-to-Video Synthesis
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Collaborative Learning for Faster StyleGAN Embedding
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Simple and Effective VAE Training with Calibrated Decoders
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• CCVS: Context-aware Controllable Video Synthesis
• FitVid: Overfitting in Pixel-Level Video Prediction
• Self-supervised Learning of Image Embedding for Continuous Control
• PreCNet: Next-Frame Video Prediction Based on Predictive Coding
• Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
• Manipulation by Feel: Touch-Based Control with Deep Predictive Models
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Predicting Video with VQVAE
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Off-Policy Evaluation via Off-Policy Classification
• CloudCast: A Satellite-Based Dataset and Baseline for Forecasting Clouds
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
• From Here to There: Video Inbetweening Using Direct 3D Convolutions
• Precipitation Nowcasting with Star-Bridge Networks
• Deep Generative Video Compression
• Taming Visually Guided Sound Generation
• CariGAN: Caricature Generation through Weakly Paired Adversarial Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Unsupervised Learning of Object Structure and Dynamics from Videos
• RoboNet: Large-Scale Multi-Robot Learning
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Experience-Embedded Visual Foresight
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Temporal Difference Variational Auto-Encoder
• PDE-Driven Spatiotemporal Disentanglement
• Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
• Procedure Planning in Instructional Videos
• DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
• Disentangling Propagation and Generation for Video Prediction
• MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
• Few-shot Video-to-Video Synthesis
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Learning Representations for Predicting Future Activities
• Order Matters: Shuffling Sequence Generation for Video Prediction
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Deep learning predictions of sand dune migration
• Hierarchical Autoregressive Modeling for Neural Video Compression
• Video Interpolation and Prediction with Unsupervised Landmarks
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Adversarial Video Compression Guided by Soft Edge Detection
• Vid2Param: Modelling of Dynamics Parameters from Video
• PISEP^2: Pseudo Image Sequence Evolution based 3D Pose Prediction
• Precipitation nowcasting using a stochastic variational frame predictor with learned prior distribution
• Compositional Video Synthesis with Action Graphs
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• World-Consistent Video-to-Video Synthesis
• Future Frame Prediction of a Video Sequence
• Deep Learned Frame Prediction for Video Compression
• Zoom-In-to-Check: Boosting Video Interpolation via Instance-level Discrimination
• Learning Disentangled Representations of Satellite Image Time Series
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders
• Understanding the Perceived Quality of Video Predictions
• Latent Neural Differential Equations for Video Generation
• Diverse Video Generation using a Gaussian Process Trigger
• Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions
• Adversarial Feature Desensitization
• Long-Term Human Video Generation of Multiple Futures Using Poses
• Motion Selective Prediction for Video Frame Synthesis
• Future Video Synthesis with Object Motion Prediction
• Hierarchical Video Generation for Complex Data
• Beta DVBF: Learning State-Space Models for Control from High Dimensional Observations
• Learning Dynamical Systems from Noisy Sensor Measurements using Multiple Shooting
• Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis
• Stochastic Dynamics for Video Infilling
• 3D Ken Burns Effect from a Single Image
• Learning to infer in recurrent biological networks
• A Foreground-background Parallel Compression with Residual Encoding for Surveillance Video
• Taylor saves for later: disentanglement for video prediction using Taylor representation
• Layered Controllable Video Generation
• A Framework for Multisensory Foresight for Embodied Agents
• VP-GO: a "light" action-conditioned visual prediction model
• Point-to-Point Video Generation
• Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure
• Hierarchical Modes Exploring in Generative Adversarial Networks
• Predicting future astronomical events using deep learning