HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN

Abstract

High-resolution video generation has emerged as a crucial task in computer vision, with wide-ranging applications in entertainment, simulation, and data augmentation. However, generating temporally coherent and visually realistic videos remains a significant challenge due to the high dimensionality and complex dynamics of video data. In this paper, we propose a novel deep generative network architecture designed specifically for high-resolution video synthesis. Our approach integrates key concepts from Wasserstein Generative Adversarial Networks (WGANs), enforcing a k-Lipschitz continuity constraint on the discriminator to stabilize training and enhance convergence. We further leverage Conditional GAN (cGAN) techniques by incorporating class labels during both training and inference, enabling class-specific video generation with improved semantic consistency. We provide a detailed layer-wise description of the Generator and Discriminator networks, highlighting architectural design choices promoting temporal coherence and spatial detail. The overall combined architecture, training algorithm, and optimization strategy are thoroughly presented. Our training objective combines a pixel-wise mean squared error loss with an adversarial loss to balance frame-level accuracy and video realism. We validate our approach on benchmark datasets including UCF101, Golf, and Aeroplane, encompassing diverse motion patterns and scene contexts. Quantitative evaluations using Inception Score (IS) and Fréchet Inception Distance (FID) demonstrate that our model significantly outperforms previous state-of-the-art unsupervised video generation methods in terms of both quality and diversity.

References in corpus (13)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Conditional Image Generation with PixelCNN Decoders
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Perceptual Adversarial Networks for Image-to-Image Transformation
• Learning to Generate Long-term Future via Hierarchical Prediction
• Adversarial Video Generation on Complex Datasets
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Bayesian Multi-Scale Neural Network for Crowd Counting