Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Abstract

The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this paper, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering.

References in corpus (50)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Generative Adversarial Networks: An Overview
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Unsupervised Learning of Video Representations using LSTMs
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Toward Multimodal Image-to-Image Translation
• The Perception-Distortion Tradeoff
• Which Training Methods for GANs do actually Converge?
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Pose Guided Person Image Generation
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Wide Activation for Efficient and Accurate Image Super-Resolution
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Unsupervised Attention-guided Image to Image Translation
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• Stochastic Adversarial Video Prediction
• Assessing Generative Models via Precision and Recall
• First Order Motion Model for Image Animation
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Adversarial Video Generation on Complex Datasets
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Diversity-Sensitive Conditional Generative Adversarial Networks
• HeadOn: Real-time Reenactment of Human Portrait Videos
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• One-Shot Unsupervised Cross Domain Translation
• Implicit Generation and Generalization in Energy-Based Models
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
• Progressive Pose Attention Transfer for Person Image Generation
• Structural-analogy from a Single Image Pair
• ReenactGAN: Learning to Reenact Faces via Boundary Transfer
• Textured Neural Avatars
• Optimal Unsupervised Domain Translation
• The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings
• Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation
• Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
• Semi-supervised Learning for Few-shot Image-to-Image Translation

Cited by in corpus (5)

• Regularizing Generative Adversarial Networks under Limited Data
• Anycost GANs for Interactive Image Synthesis and Editing
• Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data
• DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network
• Stay Positive: Non-Negative Image Synthesis for Augmented Reality