Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

Abstract

We consider the task of generating diverse and novel videos from a single video sample. Recently, new hierarchical patch-GAN based approaches were proposed for generating diverse images, given only a single sample at training time. Moving to videos, these approaches fail to generate diverse samples, and often collapse into generating samples similar to the training video. We introduce a novel patch-based variational autoencoder (VAE) which allows for a much greater diversity in generation. Using this tool, a new hierarchical video generation scheme is constructed: at coarse scales, our patch-VAE is employed, ensuring samples are of high diversity. Subsequently, at finer scales, a patch-GAN renders the fine details, resulting in high quality videos. Our experiments show that the proposed method produces diverse samples in both the image domain, and the more challenging video domain.

References in corpus (16)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Conditional Image Generation with PixelCNN Decoders
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Image Prior
• Autoencoding beyond pixels using a learned similarity metric
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Adversarial Video Generation on Complex Datasets
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs

Cited by in corpus (5)

• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting
• MOGAN: Morphologic-structure-aware Generative Learning from a Single Image
• SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs
• Meta Internal Learning