Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
arXiv:2002.09905
Abstract
Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necessity of exploring multi-frequency analysis to deal with the two problems. Inspired by the frequency band decomposition characteristic of Human Vision System (HVS), we propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner. Specifically, the multi-level spatial discrete wavelet transform decomposes each video frame into anisotropic sub-bands with multiple frequencies, helping to enrich structural information and reserve fine details. On the other hand, multi-level temporal discrete wavelet transform which operates on time axis decomposes the frame sequence into sub-band groups of different frequencies to accurately capture multi-frequency motions under a fixed frame rate. Extensive experiments on diverse datasets demonstrate that our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works.
arXiv commentAccepted by CVPR2020
References in corpus (15)
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
- A Recurrent Latent Variable Model for Sequential Data
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep multi-scale video prediction beyond mean square error
- PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
- Stochastic Video Generation with a Learned Prior
- Stochastic Adversarial Video Prediction
- Towards Accurate Generative Models of Video: A New Metric & Challenges
- Learning to Generate Long-term Future via Hierarchical Prediction
- Self-Supervised Visual Planning with Temporal Skip Connections
- Stochastic Variational Video Prediction
- Transformation-Based Models of Video Sequences
- Novel Video Prediction for Large-scale Scene using Optical Flow
- Mutual Suppression Network for Video Prediction using Disentangled Features