Future Video Synthesis with Object Motion Prediction
arXiv:2004.00542
Abstract
We present an approach to predict future video frames given a sequence of continuous video frames in the past. Instead of synthesizing images directly, our approach is designed to understand the complex scene dynamics by decoupling the background scene and moving objects. The appearance of the scene components in the future is predicted by non-rigid deformation of the background and affine transformation of moving objects. The anticipated appearances are combined to create a reasonable video in the future. With this procedure, our method exhibits much less tearing or distortion artifact compared to other approaches. Experimental results on the Cityscapes and KITTI datasets show that our model outperforms the state-of-the-art in terms of visual quality and accuracy.
arXiv commentCVPR 2020
References in corpus (14)
- Unsupervised Learning of Video Representations using LSTMs
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Stochastic Video Generation with a Learned Prior
- Spatio-temporal video autoencoder with differentiable memory
- Generative Image Inpainting with Contextual Attention
- Stochastic Adversarial Video Prediction
- Learning to Generate Long-term Future via Hierarchical Prediction
- SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks
- GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
- Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving
- Predicting Deeper into the Future of Semantic Segmentation
- Unsupervised Learning of Long-Term Motion Dynamics for Videos
- Video Frame Synthesis using Deep Voxel Flow