Flow-Grounded Spatial-Temporal Video Prediction from Still Images

Abstract

Existing video prediction methods mainly rely on observing multiple historical frames or focus on predicting the next one-frame. In this work, we study the problem of generating consecutive multiple future frames by observing one single still image only. We formulate the multi-frame prediction task as a multiple time step flow (multi-flow) prediction phase followed by a flow-to-frame synthesis phase. The multi-flow prediction is modeled in a variational probabilistic manner with spatial-temporal relationships learned through 3D convolutions. The flow-to-frame synthesis is modeled as a generative process in order to keep the predicted results lying closer to the manifold shape of real video sequence. Such a two-phase design prevents the model from directly looking at the high-dimensional pixel space of the frame sequence and is demonstrated to be more effective in predicting better and diverse results. Extensive experimental results on videos with different types of motion show that the proposed algorithm performs favorably against existing methods in terms of quality, diversity and human perceptual evaluation.

Accepted by ECCV 2018

References in corpus (7)

• Generating Videos with Scene Dynamics
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Unsupervised Learning of Disentangled Representations from Video
• Learning to Generate Long-term Future via Hierarchical Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Forecasting Human Dynamics from Static Images
• Im2Flow: Motion Hallucination from Static Images for Action Recognition

Cited by in corpus (7)

• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Predicting Video with VQVAE
• InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
• Video Generation from Single Semantic Label Map
• Few-shot Video-to-Video Synthesis
• DeepLandscape: Adversarial Modeling of Landscape Video
• G3AN: Disentangling Appearance and Motion for Video Generation