Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs

Abstract

The extension of image generation to video generation turns out to be a very difficult task, since the temporal dimension of videos introduces an extra challenge during the generation process. Besides, due to the limitation of memory and training stability, the generation becomes increasingly challenging with the increase of the resolution/duration of videos. In this work, we exploit the idea of progressive growing of Generative Adversarial Networks (GANs) for higher resolution video generation. In particular, we begin to produce video samples of low-resolution and short-duration, and then progressively increase both resolution and duration alone (or jointly) by adding new spatiotemporal convolutional layers to the current networks. Starting from the learning on a very raw-level spatial appearance and temporal movement of the video distribution, the proposed progressive method learns spatiotemporal information incrementally to generate higher resolution videos. Furthermore, we introduce a sliced version of Wasserstein GAN (SWGAN) loss to improve the distribution learning on the video data of high-dimension and mixed-spatiotemporal distribution. SWGAN loss replaces the distance between joint distributions by that of one-dimensional marginal distributions, making the loss easier to compute. We evaluate the proposed model on our collected face video dataset of 10,900 videos to generate photorealistic face videos of 256x256x32 resolution. In addition, our model also reaches a record inception score of 14.57 in unsupervised action recognition dataset UCF-101.

Master Thesis from ETH Zurich, May 22, 2018

References in corpus (2)

• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Toward Multimodal Image-to-Image Translation

Cited by in corpus (16)

• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• Lower Dimensional Kernels for Video Discriminators
• CCVS: Context-aware Controllable Video Synthesis
• InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
• Some Theoretical Insights into Wasserstein GANs
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Joint Generative and Contrastive Learning for Unsupervised Person Re-identification
• Tensorized Transformer for Dynamical Systems Modeling
• Self-Paced Video Data Augmentation with Dynamic Images Generated by Generative Adversarial Networks
• A Novel Framework for Selection of GANs for an Application
• HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN
• Layered Controllable Video Generation