Wide and Narrow: Video Prediction from Context and Motion
arXiv:2110.11586
Abstract
Video prediction, forecasting the future frames from a sequence of input frames, is a challenging task since the view changes are influenced by various factors, such as the global context surrounding the scene and local motion dynamics. In this paper, we propose a new framework to integrate these complementary attributes to predict complex pixel dynamics through deep networks. We present global context propagation networks that iteratively aggregate the non-local neighboring representations to preserve the contextual information over the past frames. To capture the local motion pattern of objects, we also devise local filter memory networks that generate adaptive filter kernels by storing the prototypical motion of moving objects in the memory. The proposed framework, utilizing the outputs from both networks, can address blurry predictions and color distortion. We conduct experiments on Caltech pedestrian and UCF101 datasets, and demonstrate state-of-the-art results. Especially for multi-step prediction, we obtain an outstanding performance in quantitative and qualitative evaluation.
arXiv commentBritish Machine Vision Conference 2021
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Unsupervised Learning of Video Representations using LSTMs
- A Review on Deep Learning Techniques for Video Prediction
- Video (language) modeling: a baseline for generative models of natural videos
- One-shot Learning with Memory-Augmented Neural Networks
- Spatio-temporal video autoencoder with differentiable memory
- Stochastic Latent Residual Video Prediction
- Kernelized Memory Network for Video Object Segmentation
- Video Extrapolation with an Invertible Linear Embedding