Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN

Abstract

Training of Generative Adversarial Network (GAN) on a video dataset is a challenge because of the sheer size of the dataset and the complexity of each observation. In general, the computational cost of training GAN scales exponentially with the resolution. In this study, we present a novel memory efficient method of unsupervised learning of high-resolution video dataset whose computational cost scales only linearly with the resolution. We achieve this by designing the generator model as a stack of small sub-generators and training the model in a specific way. We train each sub-generator with its own specific discriminator. At the time of the training, we introduce between each pair of consecutive sub-generators an auxiliary subsampling layer that reduces the frame-rate by a certain ratio. This procedure can allow each sub-generator to learn the distribution of the video at different levels of resolution. We also need only a few GPUs to train a highly complex generator that far outperforms the predecessor in terms of inception scores.

Accepted at International Journal of Computer Vision. The source code is available at https://github.com/pfnet-research/tgan2

References in corpus (38)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Residual Learning for Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Training of Wasserstein GANs
• Improved Techniques for Training GANs
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Generating Videos with Scene Dynamics
• Which Training Methods for GANs do actually Converge?
• Unsupervised Image-to-Image Translation Networks
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• cGANs with Projection Discriminator
• Deep multi-scale video prediction beyond mean square error
• FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Unsupervised Learning for Physical Interaction through Video Prediction
• Stochastic Adversarial Video Prediction
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Video-to-Video Synthesis
• Self-Supervised Visual Planning with Temporal Skip Connections
• Deep Video Generation, Prediction and Completion of Human Action Sequences
• ChainerMN: Scalable Distributed Deep Learning Framework
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
• Video Frame Synthesis using Deep Voxel Flow
• Video Pixel Networks
• Flow-Grounded Spatial-Temporal Video Prediction from Still Images
• Pose Guided Human Video Generation

Cited by in corpus (21)

• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• High Fidelity Speech Synthesis with Adversarial Networks
• ViTGAN: Training GANs with Vision Transformers
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation
• Lower Dimensional Kernels for Video Discriminators
• CCVS: Context-aware Controllable Video Synthesis
• Predicting Video with VQVAE
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• LaMD: Latent Motion Diffusion for Image-Conditional Video Generation
• Self-Paced Video Data Augmentation with Dynamic Images Generated by Generative Adversarial Networks
• Latent Neural Differential Equations for Video Generation
• Data-driven Accelerogram Synthesis using Deep Generative Models
• HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN
• Hierarchical Video Generation for Complex Data
• ArrowGAN : Learning to Generate Videos by Learning Arrow of Time
• VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs