A Good Image Generator Is What You Need for High-Resolution Video Synthesis
arXiv:2104.15069
Abstract
Image and video synthesis are closely related areas aiming at generating content from noise. While rapid progress has been demonstrated in improving image-based models to handle large resolutions, high-quality renderings, and wide variations in image content, achieving comparable video generation results remains problematic. We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. Not only does such a framework render high-resolution videos, but it also is an order of magnitude more computationally efficient. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled. With such a representation, our framework allows for a broad range of applications, including content and motion manipulation. Furthermore, we introduce a new task, which we call cross-domain video synthesis, in which the image and motion generators are trained on disjoint datasets belonging to different domains. This allows for generating moving objects for which the desired video data is not available. Extensive experiments on various datasets demonstrate the advantages of our methods over existing video generation techniques. Code will be released at https://github.com/snap-research/MoCoGAN-HD.
arXiv commentAccepted to ICLR 2021
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Improved Regularization of Convolutional Neural Networks with Cutout
- Self-Attention Generative Adversarial Networks
- VoxCeleb: a large-scale speaker identification dataset
- Learning deep representations by mutual information estimation and maximization
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Training Generative Adversarial Networks with Limited Data
- GANSpace: Discovering Interpretable GAN Controls
- Big Self-Supervised Models are Strong Semi-Supervised Learners
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Large Scale Adversarial Representation Learning
- Stochastic Video Generation with a Learned Prior
- Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
- Image Augmentations for GAN Training
- Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
- Self-Supervised Visual Planning with Temporal Skip Connections
- Differentiable Augmentation for Data-Efficient GAN Training
- Controlling generative models with continuous factors of variations
- Putting An End to End-to-End: Gradient-Isolated Learning of Representations
- Closed-Form Factorization of Latent Semantics in GANs
- Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
- Lower Dimensional Kernels for Video Discriminators
- Human Motion Transfer from Poses in the Wild