InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
arXiv:2101.03049
Abstract
In this work, we introduce an unconditional video generative model, InMoDeGAN, targeted to (a) generate high quality videos, as well as to (b) allow for interpretation of the latent space. For the latter, we place emphasis on interpreting and manipulating motion. Towards this, we decompose motion into semantic sub-spaces, which allow for control of generated samples. We design the architecture of InMoDeGAN-generator in accordance to proposed Linear Motion Decomposition, which carries the assumption that motion can be represented by a dictionary, with related vectors forming an orthogonal basis in the latent space. Each vector in the basis represents a semantic sub-space. In addition, a Temporal Pyramid Discriminator analyzes videos at different temporal resolutions. Extensive quantitative and qualitative analysis shows that our model systematically and significantly outperforms state-of-the-art methods on the VoxCeleb2-mini and BAIR-robot datasets w.r.t. video quality related to (a). Towards (b) we present experimental results, confirming that decomposed sub-spaces are interpretable and moreover, generated motion is controllable.
Please visit https://wyhsirius.github.io/InMoDeGAN/ for introductions and more
References in corpus (8)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Understanding the Role of Individual Units in a Deep Neural Network
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- Self-Supervised Visual Planning with Temporal Skip Connections
- Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture
- Video Generation from Single Semantic Label Map