Video Generation from Single Semantic Label Map
arXiv:1903.04480
Abstract
This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation process. Different from typical end-to-end approaches, which model both scene content and dynamics in a single step, we propose to decompose this difficult task into two sub-problems. As current image generation methods do better than video generation in terms of detail, we synthesize high quality content by only generating the first frame. Then we animate the scene based on its semantic meaning to obtain the temporally coherent video, giving us excellent results overall. We employ a cVAE for predicting optical flow as a beneficial intermediate step to generate a video sequence conditioned on the initial single frame. A semantic label map is integrated into the flow prediction module to achieve major improvements in the image-to-video generation process. Extensive experiments on the Cityscapes dataset show that our method outperforms all competing methods.
Paper accepted at CVPR 2019. Source code and models available at https://github.com/junting/seg2vid/tree/master
References in corpus (8)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Unsupervised Learning of Video Representations using LSTMs
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Video (language) modeling: a baseline for generative models of natural videos
- Video-to-Video Synthesis
- Stochastic Variational Video Prediction
- Flow-Grounded Spatial-Temporal Video Prediction from Still Images
Cited by in corpus (7)
- CCVS: Context-aware Controllable Video Synthesis
- InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
- Few-shot Video-to-Video Synthesis
- G3AN: Disentangling Appearance and Motion for Video Generation
- Cross-View Exocentric to Egocentric Video Synthesis
- Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis
- Layered Controllable Video Generation