MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

Abstract

This paper tackles video prediction from a new dimension of predicting spacetime-varying motions that are incessantly changing across both space and time. Prior methods mainly capture the temporal state transitions but overlook the complex spatiotemporal variations of the motion itself, making them difficult to adapt to ever-changing motions. We observe that physical world motions can be decomposed into transient variation and motion trend, while the latter can be regarded as the accumulation of previous motions. Thus, simultaneously capturing the transient variation and the motion trend is the key to make spacetime-varying motions more predictable. Based on these observations, we propose the MotionRNN framework, which can capture the complex variations within motions and adapt to spacetime-varying scenarios. MotionRNN has two main contributions. The first is that we design the MotionGRU unit, which can model the transient variation and motion trend in a unified way. The second is that we apply the MotionGRU to RNN-based predictive models and indicate a new flexible video prediction architecture with a Motion Highway that can significantly improve the ability to predict changeable motions and avoid motion vanishing for stacked multiple-layer predictive models. With high flexibility, this framework can adapt to a series of models for deterministic spatiotemporal prediction. Our MotionRNN can yield significant improvements on three challenging benchmarks for video prediction with spacetime-varying motions.

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Unsupervised Learning of Video Representations using LSTMs
• Training Very Deep Networks
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Stochastic Adversarial Video Prediction
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Stochastic Latent Residual Video Prediction