Adversarial Video Generation on Complex Datasets

Abstract

Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• The Kinetics Human Action Video Dataset
• Self-Attention Generative Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Learned Representation For Artistic Style
• Generating Long Sequences with Sparse Transformers
• cGANs with Projection Discriminator
• Deep multi-scale video prediction beyond mean square error
• Stochastic Video Generation with a Learned Prior
• A Short Note about Kinetics-600
• Modulating early visual processing by language
• Video (language) modeling: a baseline for generative models of natural videos
• A Note on the Inception Score
• Stochastic Adversarial Video Prediction
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Learning to Generate Long-term Future via Hierarchical Prediction
• Assessing Generative Models via Precision and Recall
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Video-to-Video Synthesis
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Self-Supervised Visual Planning with Temporal Skip Connections
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Geometric GAN
• On Multiplicative Integration with Recurrent Neural Networks
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
• TF-Replicator: Distributed Machine Learning for Researchers
• Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture
• Scaling Autoregressive Video Models
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos

Cited by in corpus (30)

• Skillful Precipitation Nowcasting using Deep Generative Models of Radar
• A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions
• A Review on Deep Learning Techniques for Video Prediction
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• VideoGPT: Video Generation using VQ-VAE and Transformers
• High Fidelity Speech Synthesis with Adversarial Networks
• Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN)
• Behavioral Use Licensing for Responsible AI
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• CCVS: Context-aware Controllable Video Synthesis
• FitVid: Overfitting in Pixel-Level Video Prediction
• Predicting Video with VQVAE
• Encoding large scale cosmological structure with Generative Adversarial Networks
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Clockwork Variational Autoencoders
• Understanding Overparameterization in Generative Adversarial Networks
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Generative modeling of spatio-temporal weather patterns with extreme event conditioning
• A Novel Framework for Selection of GANs for an Application
• Latent Neural Differential Equations for Video Generation
• NWT: Towards natural audio-to-video generation with representation learning
• A Survey on GAN Acceleration Using Memory Compression Technique
• HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN
• Hierarchical Video Generation for Complex Data
• NeurInt : Learning to Interpolate through Neural ODEs
• Xp-GAN: Unsupervised Multi-object Controllable Video Generation
• Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers