Stochastic Dynamics for Video Infilling

Abstract

In this paper, we introduce a stochastic dynamics video infilling (SDVI) framework to generate frames between long intervals in a video. Our task differs from video interpolation which aims to produce transitional frames for a short interval between every two frames and increase the temporal resolution. Our task, namely video infilling, however, aims to infill long intervals with plausible frame sequences. Our framework models the infilling as a constrained stochastic generation process and sequentially samples dynamics from the inferred distribution. SDVI consists of two parts: (1) a bi-directional constraint propagation module to guarantee the spatial-temporal coherence among frames, (2) a stochastic sampling process to generate dynamics from the inferred distributions. Experimental results show that SDVI can generate clear frame sequences with varying contents. Moreover, motions in the generated sequence are realistic and able to transfer smoothly from the given start frame to the terminal frame. Our project site is https://xharlie.github.io/projects/project_sites/SDVI/video_results.html

arXiv commentWinter Conference on Applications of Computer Vision (WACV 2020)

References in corpus (16)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Unsupervised Learning of Video Representations using LSTMs
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep multi-scale video prediction beyond mean square error
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• Self-Supervised Visual Planning with Temporal Skip Connections
• Hierarchical Long-term Video Prediction without Supervision
• Stochastic Variational Video Prediction
• Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
• Context-aware Synthesis for Video Frame Interpolation
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Prediction Under Uncertainty with Error-Encoding Networks