Visual Forecasting by Imitating Dynamics in Natural Sequences
arXiv:1708.05827
Abstract
We introduce a general framework for visual forecasting, which directly imitates visual sequences without additional supervision. As a result, our model can be applied at several semantic levels and does not require any domain knowledge or handcrafted features. We achieve this by formulating visual forecasting as an inverse reinforcement learning (IRL) problem, and directly imitate the dynamics in natural sequences from their raw pixel values. The key challenge is the high-dimensional and continuous state-action space that prohibits the application of previous IRL algorithms. We address this computational bottleneck by extending recent progress in model-free imitation with trainable deep feature representations, which (1) bypasses the exhaustive state-action pair visits in dynamic programming by using a dual formulation and (2) avoids explicit state sampling at gradient computation using a deep feature reparametrization. This allows us to apply IRL at scale and directly imitate the dynamics in high-dimensional continuous visual sequences from the raw pixel values. We evaluate our approach at three different level-of-abstraction, from low level pixels to higher level semantics: future frame generation, action anticipation, visual story forecasting. At all levels, our approach outperforms existing methods.
10 pages, 9 figures, accepted to ICCV 2017
References in corpus (18)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Deep Residual Learning for Image Recognition
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Generating Videos with Scene Dynamics
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Generative Adversarial Imitation Learning
- SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Unsupervised Learning for Physical Interaction through Video Prediction
- Adversarial Learning for Neural Dialogue Generation
- Unsupervised Learning of Visual Representations using Videos
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Visual Storytelling
- Attribute2Image: Conditional Image Generation from Visual Attributes
- Model-Free Imitation Learning with Policy Optimization
- Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
- Learning Features by Watching Objects Move
- Learning Visual Storylines with Skipping Recurrent Neural Networks
- Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization