InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
arXiv:2101.03049
Abstract
In this work, we introduce an unconditional video generative model, InMoDeGAN, targeted to (a) generate high quality videos, as well as to (b) allow for interpretation of the latent space. For the latter, we place emphasis on interpreting and manipulating motion. Towards this, we decompose motion into semantic sub-spaces, which allow for control of generated samples. We design the architecture of InMoDeGAN-generator in accordance to proposed Linear Motion Decomposition, which carries the assumption that motion can be represented by a dictionary, with related vectors forming an orthogonal basis in the latent space. Each vector in the basis represents a semantic sub-space. In addition, a Temporal Pyramid Discriminator analyzes videos at different temporal resolutions. Extensive quantitative and qualitative analysis shows that our model systematically and significantly outperforms state-of-the-art methods on the VoxCeleb2-mini and BAIR-robot datasets w.r.t. video quality related to (a). Towards (b) we present experimental results, confirming that decomposed sub-spaces are interpretable and moreover, generated motion is controllable.
Please visit https://wyhsirius.github.io/InMoDeGAN/ for introductions and more
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Learning from Synthetic Humans
- Which Training Methods for GANs do actually Converge?
- Understanding the Role of Individual Units in a Deep Neural Network
- Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- Self-Supervised Visual Planning with Temporal Skip Connections
- Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
- Video Prediction with Appearance and Motion Conditions
- Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture
- Video Generation from Single Semantic Label Map
- Flow-Grounded Spatial-Temporal Video Prediction from Still Images
- Learning to Forecast and Refine Residual Motion for Image-to-Video Generation