Learning Semantic-Aware Dynamics for Video Prediction
arXiv:2104.09762
Abstract
We propose an architecture and training scheme to predict video frames by explicitly modeling dis-occlusions and capturing the evolution of semantically consistent regions in the video. The scene layout (semantic map) and motion (optical flow) are decomposed into layers, which are predicted and fused with their context to generate future layouts and motions. The appearance of the scene is warped from past frames using the predicted motion in co-visible regions; dis-occluded regions are synthesized with content-aware inpainting utilizing the predicted scene layout. The result is a predictive model that explicitly represents objects and learns their class-specific motion, which we evaluate on video prediction benchmarks.
arXiv commentPaper accepted at CVPR 2021
References in corpus (8)
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Learning to Generate Long-term Future via Hierarchical Prediction
- Video-to-Video Synthesis
- Stochastic Variational Video Prediction
- Deep Flow-Guided Video Inpainting