TwoStreamVAN: Improving Motion Modeling in Video Generation
arXiv:1812.01037
Abstract
Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content. Existing methods entangle the two intrinsically different tasks of motion and content creation in a single generator network, but this approach struggles to simultaneously generate plausible motion and content. To im-prove motion modeling in video generation tasks, we propose a two-stream model that disentangles motion generation from content generation, called a Two-Stream Variational Adversarial Network (TwoStreamVAN). Given an action label and a noise vector, our model is able to create clear and consistent motion, and thus yields photorealistic videos. The key idea is to progressively generate and fuse multi-scale motion with its corresponding spatial content. Our model significantly outperforms existing methods on the standard Weizmann Human Action, MUG Facial Expression, and VoxCeleb datasets, as well as our new dataset of diverse human actions with challenging and complex motion. Our code is available at https://github.com/sunxm2357/TwoStreamVAN/.
arXiv commentWACV 2020
References in corpus (13)
- Auto-Encoding Variational Bayes
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Generative Adversarial Text to Image Synthesis
- VoxCeleb: a large-scale speaker identification dataset
- Conditional Image Synthesis With Auxiliary Classifier GANs
- Unsupervised Learning of Video Representations using LSTMs
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Stochastic Video Generation with a Learned Prior
- On Convergence and Stability of GANs
- Stochastic Adversarial Video Prediction
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- PixelVAE: A Latent Variable Model for Natural Images