VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
arXiv:1903.01434
Abstract
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modelling of video.
arXiv commentICLR 2020 Camera-Ready. Previous title: VideoFlow: A Flow-Based Generative Model for Video
References in corpus (19)
- Auto-Encoding Variational Bayes
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Playing Atari with Deep Reinforcement Learning
- WaveNet: A Generative Model for Raw Audio
- Conditional Image Generation with PixelCNN Decoders
- Unsupervised Learning of Video Representations using LSTMs
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Density estimation using Real NVP
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Stochastic Video Generation with a Learned Prior
- Video (language) modeling: a baseline for generative models of natural videos
- Stochastic Adversarial Video Prediction
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Self-Supervised Visual Planning with Temporal Skip Connections
Cited by in corpus (11)
- CCVS: Context-aware Controllable Video Synthesis
- Predicting Video with VQVAE
- VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
- Telescoping Density-Ratio Estimation
- Future Frame Prediction for Robot-assisted Surgery
- OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
- Robust model training and generalisation with Studentising flows
- Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure
- Layered Controllable Video Generation
- CGAP2: Context and gap aware predictive pose framework for early detection of gestures
- Learning to Transfer Visual Effects from Videos to Images