Learning to Forecast Videos of Human Activity with Multi-granularity Models and Adaptive Rendering
arXiv:1712.01955
Abstract
We propose an approach for forecasting video of complex human activity involving multiple people. Direct pixel-level prediction is too simple to handle the appearance variability in complex activities. Hence, we develop novel intermediate representations. An architecture combining a hierarchical temporal model for predicting human poses and encoder-decoder convolutional neural networks for rendering target appearances is proposed. Our hierarchical model captures interactions among people by adopting a dynamic group-based interaction mechanism. Next, our appearance rendering network encodes the targets' appearances by learning adaptive appearance filters using a fully convolutional network. Finally, these filters are placed in encoder-decoder neural networks to complete the rendering. We demonstrate that our model can generate videos that are superior to state-of-the-art methods, and can handle complex human activity scenarios in video forecasting.
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Categorical Reparameterization with Gumbel-Softmax
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Deep multi-scale video prediction beyond mean square error
- Video (language) modeling: a baseline for generative models of natural videos
- Unsupervised Learning for Physical Interaction through Video Prediction
- Learning to Generate Long-term Future via Hierarchical Prediction
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- StyleBank: An Explicit Representation for Neural Image Style Transfer
- Generating Images from Captions with Attention
- Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition