A Review on Deep Learning Techniques for Video Prediction

Abstract

The ability to predict, anticipate and reason about future outcomes is a key component of intelligent decision-making systems. In light of the success of deep learning in computer vision, deep-learning-based video prediction emerged as a promising research direction. Defined as a self-supervised learning task, video prediction represents a suitable framework for representation learning, as it demonstrated potential capabilities for extracting meaningful representations of the underlying patterns in natural videos. Motivated by the increasing interest in this task, we provide a review on the deep learning methods for prediction in video sequences. We firstly define the video prediction fundamentals, as well as mandatory background concepts and the most used datasets. Next, we carefully analyze existing video prediction models organized according to a proposed taxonomy, highlighting their contributions and their significance in the field. The summary of the datasets and methods is accompanied with experimental results that facilitate the assessment of the state of the art on a quantitative basis. The paper is summarized by drawing some general conclusions, identifying open research challenges and by pointing out future research directions.

Submitted to TPAMI

References in corpus (79)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Conditional Generative Adversarial Nets
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Conditional Image Generation with PixelCNN Decoders
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• YFCC100M: The New Data in Multimedia Research
• Pixel Recurrent Neural Networks
• Do Deep Nets Really Need to be Deep?
• NICE: Non-linear Independent Components Estimation
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Generating Videos with Scene Dynamics
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review
• A Recurrent Latent Variable Model for Sequential Data
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
• Dynamic Filter Networks
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Decomposing Motion and Content for Natural Video Sequence Prediction
• A note on the evaluation of generative models
• PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
• Stochastic Video Generation with a Learned Prior
• A Short Note about Kinetics-600
• Video (language) modeling: a baseline for generative models of natural videos
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Ladder Variational Autoencoders
• Spatio-temporal video autoencoder with differentiable memory
• Unsupervised Learning of Disentangled Representations from Video
• Learning What and Where to Draw
• Stochastic Adversarial Video Prediction
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Learning a Driving Simulator
• Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
• Adversarial Video Generation on Complex Datasets
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Human Action Recognition and Prediction: A Survey
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Recurrent Environment Simulators
• Learning to Decompose and Disentangle Representations for Video Prediction
• Learning Visual Predictive Models of Physics for Playing Billiards
• Hierarchical Long-term Video Prediction without Supervision
• Unsupervised Learning of Visual Structure using Predictive Generative Networks
• Learning to Act by Predicting the Future
• Transformation-Based Models of Video Sequences
• On Loss Functions for Deep Neural Networks in Classification
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Predicting Scene Parsing and Motion Dynamics in the Future
• Multi-Dimensional Recurrent Neural Networks
• Long-term Planning by Short-term Prediction
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• Deep Predictive Coding Networks
• Learning to Linearize Under Uncertainty
• Future Semantic Segmentation with Convolutional LSTM
• Video Imagination from a Single Image with Transformation Generation
• Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames
• Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods
• Video Ladder Networks
• RoboNet: Large-Scale Multi-Robot Learning
• Prediction Under Uncertainty with Error-Encoding Networks
• Motion Prediction Under Multimodality with Conditional Stochastic Networks
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Mutual Suppression Network for Video Prediction using Disentangled Features
• Recurrent Ladder Networks
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Future Segmentation Using 3D Structure
• Video Extrapolation with an Invertible Linear Embedding
• FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs
• Long-Term Human Video Generation of Multiple Futures Using Poses

Cited by in corpus (23)

• Video Transformers: A Survey
• Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
• Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directions
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• FitVid: Overfitting in Pixel-Level Video Prediction
• Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
• A Predictive Approach To Enhance Time-Series Forecasting
• GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis
• Performing Video Frame Prediction of Microbial Growth with a Recurrent Neural Network
• Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion
• Meta-Learning Dynamics Forecasting Using Task Inference
• Forecasting Future Instance Segmentation with Learned Optical Flow and Warping
• Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images
• Deep Video Prediction for Time Series Forecasting
• Structural Forecasting for Tropical Cyclone Intensity Prediction: Providing Insight with Deep Learning
• Understanding the Perceived Quality of Video Predictions
• Wide and Narrow: Video Prediction from Context and Motion
• DFPN: Deformable Frame Prediction Network
• Reflections on "Can AI Understand Our Universe?"
• Predicting future astronomical events using deep learning
• Towards an Interpretable Latent Space in Structured Models for Video Prediction
• From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting
• FREGAN : an application of generative adversarial networks in enhancing the frame rate of videos