PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

Abstract

We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal memories in a cascaded mechanism. However, there is still a dilemma in video predictive learning: increasingly deep-in-time models have been designed for capturing complex variations, while introducing more difficulties in the gradient back-propagation. To alleviate this undesirable effect, we propose a Gradient Highway architecture, which provides alternative shorter routes for gradient flows from outputs back to long-range inputs. This architecture works seamlessly with causal LSTMs, enabling PredRNN++ to capture short-term and long-term dependencies adaptively. We assess our model on both synthetic and real video datasets, showing its ability to ease the vanishing gradient problem and yield state-of-the-art prediction results even in a difficult objects occlusion scenario.

Cited by in corpus (38)

• A Review on Deep Learning Techniques for Video Prediction
• Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Machine Learning for Spatiotemporal Sequence Forecasting: A Survey
• Learning Dynamical Systems from Partial Observations
• Advancing Learned Video Compression with In-loop Frame Prediction
• Comparing recurrent and convolutional neural networks for predicting wave propagation
• Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning
• CloudCast: A Satellite-Based Dataset and Baseline for Forecasting Clouds
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
• TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regression
• Precipitation Nowcasting with Star-Bridge Networks
• PredNet and Predictive Coding: A Critical Review
• Reduced-Gate Convolutional LSTM Using Predictive Coding for Spatiotemporal Prediction
• Unsupervised Learning from Video with Deep Neural Embeddings
• PDE-Driven Spatiotemporal Disentanglement
• SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation
• StressNet: Deep Learning to Predict Stress With Fracture Propagation in Brittle Materials
• CrevNet: Conditionally Reversible Video Prediction
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Deep Vision in Analysis and Recognition of Radar Data: Achievements, Advancements and Challenges
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Active World Model Learning with Progress Curiosity
• PISEP^2: Pseudo Image Sequence Evolution based 3D Pose Prediction
• Frequency Domain Transformer Networks for Video Prediction
• Learning to Transfer for Traffic Forecasting via Multi-task Learning
• Neural Allocentric Intuitive Physics Prediction from Real Videos
• Variable Rate Video Compression using a Hybrid Recurrent Convolutional Learning Framework
• Modular Action Concept Grounding in Semantic Video Prediction
• Learning Disentangled Representations of Video with Missing Data
• Rain-Code Fusion : Code-to-code ConvLSTM Forecasting Spatiotemporal Precipitation
• Learning Scene Dynamics from Point Cloud Sequences
• Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos
• Multimodal fusion for sea level anomaly forecasting
• Capturing Evolution Genes for Time Series Data
• Predictive Process Model Monitoring using Recurrent Neural Networks
• Space Time Recurrent Memory Network
• From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting