Decomposing Motion and Content for Natural Video Sequence Prediction
arXiv:1706.08033
Abstract
We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.
International Conference on Learning Representations (ICLR) 2017
References in corpus (2)
Cited by in corpus (108)
- Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
- Recent Advances in Autoencoder-Based Representation Learning
- A Review on Deep Learning Techniques for Video Prediction
- Multimodal Unsupervised Image-to-Image Translation
- Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
- A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
- Unsupervised Scalable Representation Learning for Multivariate Time Series
- Stochastic Adversarial Video Prediction
- Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
- Dynamical Variational Autoencoders: A Comprehensive Review
- Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
- Video-to-Video Synthesis
- Learning to Decompose and Disentangle Representations for Video Prediction
- Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
- Convolutional Tensor-Train LSTM for Spatio-temporal Learning
- Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
- Machine Learning for Spatiotemporal Sequence Forecasting: A Survey
- A Good Image Generator Is What You Need for High-Resolution Video Synthesis
- Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets
- ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
- Small Sample Learning in Big Data Era
- Symbolic Pregression: Discovering Physical Laws from Distorted Video
- Detecting Adversarial Attacks on Neural Network Policies with Visual Foresight
- Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
- Future Semantic Segmentation with Convolutional LSTM
- VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
- Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
- Motion-Appearance Co-Memory Networks for Video Question Answering
- Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
- Talking Face Generation by Conditional Recurrent Adversarial Network
- Dual Contrastive Learning for Spatio-temporal Representation
- Learning Disentangled Representations with Reference-Based Variational Autoencoders
- A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception
- Unsupervised Learning of Object Structure and Dynamics from Videos
- Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
- Inception-inspired LSTM for Next-frame Video Prediction
- Prediction Under Uncertainty with Error-Encoding Networks
- Disentangled State Space Representations
- Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
- Video Generation from Single Semantic Label Map
- Transformation-based Adversarial Video Prediction on Large-Scale Data
- PDE-Driven Spatiotemporal Disentanglement
- Mutual Suppression Network for Video Prediction using Disentangled Features
- Flow-Grounded Spatial-Temporal Video Prediction from Still Images
- TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model
- Disentangling Propagation and Generation for Video Prediction
- MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
- SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
- Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data
- S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
- Learning to Forecast and Refine Residual Motion for Image-to-Video Generation
- Few-shot Video-to-Video Synthesis
- Order Matters: Shuffling Sequence Generation for Video Prediction
- Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
- Anomaly Detection using Deep Reconstruction and Forecasting for Autonomous Systems
- TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting
- Self-Supervised Learning via multi-Transformation Classification for Action Recognition
- Meta-Learning Dynamics Forecasting Using Task Inference
- Deep RNN Framework for Visual Sequential Applications
- Learning World Transition Model for Socially Aware Robot Navigation
- Independent Innovation Analysis for Nonlinear Vector Autoregressive Process
- PISEP^2: Pseudo Image Sequence Evolution based 3D Pose Prediction
- Measuring the Biases and Effectiveness of Content-Style Disentanglement
- Multi-View Frame Reconstruction with Conditional GAN
- Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction
- FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting
- Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
- Future Segmentation Using 3D Structure
- Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images
- Predicting tongue motion in unlabeled ultrasound videos using convolutional LSTM neural network
- Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos
- Video Reenactment as Inductive Bias for Content-Motion Disentanglement
- Disentangling Motion, Foreground and Background Features in Videos
- A Temporally-Aware Interpolation Network for Video Frame Inpainting
- Understanding the Perceived Quality of Video Predictions
- Object Localization with a Weakly Supervised CapsNet
- Learning Temporal Dynamics from Cycles in Narrated Video
- Modular Action Concept Grounding in Semantic Video Prediction
- Future Frame Prediction for Robot-assisted Surgery
- Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning
- Learning Disentangled Representations of Video with Missing Data
- Exploring Self-Attention for Visual Odometry
- Predicting the Future with Transformational States
- Inserting Videos into Videos
- Generative Spatiotemporal Modeling Of Neutrophil Behavior
- Motion Selective Prediction for Video Frame Synthesis
- Hierarchical Video Generation for Complex Data
- Recurrent Flow-Guided Semantic Forecasting
- Causal Future Prediction in a Minkowski Space-Time
- Sound2Sight: Generating Visual Dynamics from Sound and Context
- DeepLandscape: Adversarial Modeling of Landscape Video
- Multi-Frame Content Integration with a Spatio-Temporal Attention Mechanism for Person Video Motion Transfer
- Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
- From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting
- Deep Variational Luenberger-type Observer for Stochastic Video Prediction
- FREGAN : an application of generative adversarial networks in enhancing the frame rate of videos
- Visual Reaction: Learning to Play Catch with Your Drone
- Generating Videos of Zero-Shot Compositions of Actions and Objects
- Self-Supervised Equivariant Scene Synthesis from Video
- Compositional Video Prediction
- Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis
- Point-to-Point Video Generation
- Learning Semantic-Aware Dynamics for Video Prediction
- SDCNet: Video Prediction Using Spatially-Displaced Convolution
- Stochastic Dynamics for Video Infilling
- Towards Purely Unsupervised Disentanglement of Appearance and Shape for Person Images Generation
- MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
- Cubic LSTMs for Video Prediction