Stochastic Video Generation with a Learned Prior

Abstract

Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry generations, or both. In this paper we introduce an unsupervised video generation model that learns a prior model of uncertainty in a given environment. Video frames are generated by drawing samples from this prior and combining them with a deterministic estimate of the future frame. The approach is simple and easily trained end-to-end on a variety of datasets. Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.

Cited by in corpus (107)

• A Review on Deep Learning Techniques for Video Prediction
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Learning to Predict the Cosmological Structure Formation
• Stochastic Adversarial Video Prediction
• Dynamical Variational Autoencoders: A Comprehensive Review
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Adversarial Video Generation on Complex Datasets
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Fashion-Gen: The Generative Fashion Dataset and Challenge
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Machine Learning for Spatiotemporal Sequence Forecasting: A Survey
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• A Novel Graph based Trajectory Predictor with Pseudo Oracle
• Recurrent Independent Mechanisms
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• A review of radar-based nowcasting of precipitation and applicable machine learning techniques
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Simple and Effective VAE Training with Calibrated Decoders
• CCVS: Context-aware Controllable Video Synthesis
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• FitVid: Overfitting in Pixel-Level Video Prediction
• PreCNet: Next-Frame Video Prediction Based on Predictive Coding
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Predicting Video with VQVAE
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Mastering Atari with Discrete World Models
• StoryGAN: A Sequential Conditional GAN for Story Visualization
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
• CloudCast: A Satellite-Based Dataset and Baseline for Forecasting Clouds
• High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
• From Here to There: Video Inbetweening Using Direct 3D Convolutions
• Deep Generative Video Compression
• Unsupervised Learning of Object Structure and Dynamics from Videos
• Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Structured Variational Inference for Simulating Populations of Radio Galaxies
• Contrastively Disentangled Sequential Variational Autoencoder
• Scaling Autoregressive Video Models
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Preventing Posterior Collapse with delta-VAEs
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Clockwork Variational Autoencoders
• PDE-Driven Spatiotemporal Disentanglement
• Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
• Disentangling Propagation and Generation for Video Prediction
• MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
• TPPO: A Novel Trajectory Predictor with Pseudo Oracle
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Pedestrian Wind Factor Estimation in Complex Urban Environments
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Hierarchical Autoregressive Modeling for Neural Video Compression
• Noether Networks: Meta-Learning Useful Conserved Quantities
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Model-based Behavioral Cloning with Future Image Similarity Learning
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Video Interpolation and Prediction with Unsupervised Landmarks
• Adversarial Video Compression Guided by Soft Edge Detection
• Learning to Take Directions One Step at a Time
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Compositional Video Synthesis with Action Graphs
• Simple Video Generation using Neural ODEs
• Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Variational Tracking and Prediction with Generative Disentangled State-Space Models
• Deep Learned Frame Prediction for Video Compression
• Prediction-assistant Frame Super-Resolution for Video Streaming
• How to make a pizza: Learning a compositional layer-based GAN model
• Latent Neural Differential Equations for Video Generation
• Methods for Recovering Conditional Independence Graphs: A Survey
• Learning Disentangled Representations of Video with Missing Data
• Understanding the Perceived Quality of Video Predictions
• Future Frame Prediction for Robot-assisted Surgery
• Controllable Image Synthesis via SegVAE
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
• Modular Action Concept Grounding in Semantic Video Prediction
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Hierarchical Video Generation for Complex Data
• Future Video Synthesis with Object Motion Prediction
• Sound2Sight: Generating Visual Dynamics from Sound and Context
• Motion Selective Prediction for Video Frame Synthesis
• ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning
• Towards Good Practices of U-Net for Traffic Forecasting
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• Layered Controllable Video Generation
• Taylor saves for later: disentanglement for video prediction using Taylor representation
• Action2video: Generating Videos of Human 3D Actions
• SDCNet: Video Prediction Using Spatially-Displaced Convolution
• Hierarchical Motion Understanding via Motion Programs
• A Variational Auto-Encoder Model for Stochastic Point Processes
• Online Unsupervised Learning of Visual Representations and Categories
• VP-GO: a "light" action-conditioned visual prediction model
• Stochastic Dynamics for Video Infilling
• ArrowGAN : Learning to Generate Videos by Learning Arrow of Time
• Point-to-Point Video Generation
• Compositional Video Prediction
• From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting
• Disentangled Dynamic Representations from Unordered Data
• Cross-View Exocentric to Egocentric Video Synthesis
• Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure