Taylor saves for later: disentanglement for video prediction using Taylor representation

Abstract

Video prediction is a challenging task with wide application prospects in meteorology and robot systems. Existing works fail to trade off short-term and long-term prediction performances and extract robust latent dynamics laws in video frames. We propose a two-branch seq-to-seq deep model to disentangle the Taylor feature and the residual feature in video frames by a novel recurrent prediction module (TaylorCell) and residual module. TaylorCell can expand the video frames' high-dimensional features into the finite Taylor series to describe the latent laws. In TaylorCell, we propose the Taylor prediction unit (TPU) and the memory correction unit (MCU). TPU employs the first input frame's derivative information to predict the future frames, avoiding error accumulation. MCU distills all past frames' information to correct the predicted Taylor feature from TPU. Correspondingly, the residual module extracts the residual feature complementary to the Taylor feature. On three generalist datasets (Moving MNIST, TaxiBJ, Human 3.6), our model outperforms or reaches state-of-the-art models, and ablation experiments demonstrate the effectiveness of our model in long-term prediction.

References in corpus (10)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Learning of Video Representations using LSTMs
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• Learning to Generate Long-term Future via Hierarchical Prediction
• Stochastic Latent Residual Video Prediction
• Video Prediction via Example Guidance
• Discriminating Spatial and Temporal Relevance in Deep Taylor Decompositions for Explainable Activity Recognition