Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy

Abstract

Generative adversarial networks (GANs) have been extensively studied in the past few years. Arguably their most significant impact has been in the area of computer vision where great advances have been made in challenges such as plausible image generation, image-to-image translation, facial attribute manipulation and similar domains. Despite the significant successes achieved to date, applying GANs to real-world problems still poses significant challenges, three of which we focus on here. These are: (1) the generation of high quality images, (2) diversity of image generation, and (3) stable training. Focusing on the degree to which popular GAN technologies have made progress against these challenges, we provide a detailed review of the state of the art in GAN-related research in the published scientific literature. We further structure this review through a convenient taxonomy we have adopted based on variations in GAN architectures and loss functions. While several reviews for GANs have been presented to date, none have considered the status of this field based on their progress towards addressing practical challenges relevant to computer vision. Accordingly, we review and critically discuss the most popular architecture-variant, and loss-variant GANs, for tackling these challenges. Our objective is to provide an overview as well as a critical analysis of the status of GAN research in terms of relevant progress towards important computer vision application requirements. As we do this we also discuss the most compelling applications in computer vision in which GANs have demonstrated considerable success along with some suggestions for future research directions. Code related to GAN-variants studied in this work is summarized on https://github.com/sheqi/GAN_Review.

Accepted by ACM Computing Surveys, 23 November 2020

References in corpus (56)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Generative Adversarial Networks: An Overview
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• MesoNet: a Compact Facial Video Forgery Detection Network
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Tutorial on Variational Autoencoders
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• NIPS 2016 Tutorial: Generative Adversarial Networks
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Toward Multimodal Image-to-Image Translation
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Adversarial Feature Learning
• Pose Guided Person Image Generation
• The relativistic discriminator: a key element missing from standard GAN
• Semi-Supervised Learning with Generative Adversarial Networks
• Semantic Segmentation using Adversarial Networks
• A note on the evaluation of generative models
• Deep Reinforcement Learning for Dialogue Generation
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Generative Moment Matching Networks
• Towards Principled Methods for Training Generative Adversarial Networks
• MaskGAN: Better Text Generation via Filling in the______
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• Neural Photo Editing with Introspective Adversarial Networks
• Mode Regularized Generative Adversarial Networks
• On Convergence and Stability of GANs
• An empirical study on evaluation metrics of generative adversarial networks
• An Actor-Critic Algorithm for Sequence Prediction
• A Note on the Inception Score
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Generative Image Inpainting with Contextual Attention
• Adversarial Video Generation on Complex Datasets
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
• Texture Synthesis with Spatial Generative Adversarial Networks
• On integral probability metrics, ϕ-divergences and binary classification
• GP-GAN: Towards Realistic High-Resolution Image Blending
• Improved generator objectives for GANs
• Adversarial Deep Structural Networks for Mammographic Mass Segmentation
• Lower Dimensional Kernels for Video Discriminators
• High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks
• Synthesis of Realistic ECG using Generative Adversarial Networks
• Quick and Easy Time Series Generation with Established Image-based GANs
• AlphaGAN: Generative adversarial networks for natural image matting
• Comparative Study on Generative Adversarial Networks
• Real or Not Real, that is the Question
• Quantitatively Evaluating GANs With Divergences Proposed for Training

Cited by in corpus (41)

• Deep Neural Networks and Tabular Data: A Survey
• Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Medical Deep Learning -- A systematic Meta-Review
• Adversarial Text-to-Image Synthesis: A Review
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• Generative Adversarial Networks for Spatio-temporal Data: A Survey
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Review: Deep Learning in Electron Microscopy
• Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging
• QuGAN: A Quantum State Fidelity based Generative Adversarial Network
• Hyperspectral Denoising Using Unsupervised Disentangled Spatio-Spectral Deep Priors
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Synthetic flow-based cryptomining attack generation through Generative Adversarial Networks
• Bi-fidelity Variational Auto-encoder for Uncertainty Quantification
• The Six Fronts of the Generative Adversarial Networks
• Material Measurement Units for a Circular Economy: Foundations through a Review
• GAN-based Tabular Data Generator for Constructing Synopsis in Approximate Query Processing: Challenges and Solutions
• Improving the quality of generative models through Smirnov transformation
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
• Feasibility of Modeling Orthogonal Frequency-Division Multiplexing Communication Signals with Unsupervised Generative Adversarial Networks
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue
• AVGCN: Trajectory Prediction using Graph Convolutional Networks Guided by Human Attention
• Digital Shielding for Cross-Domain Wi-Fi Signal Adaptation using Relativistic Average Generative Adversarial Network
• Pores for thought: The use of generative adversarial networks for the stochastic reconstruction of 3D multi-phase electrode microstructures with periodic boundaries
• A Novel Framework for Selection of GANs for an Application
• Adaptive Input-image Normalization for Solving the Mode Collapse Problem in GAN-based X-ray Images
• A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition
• IROS 2019 Lifelong Robotic Vision Challenge -- Lifelong Object Recognition Report
• A Generative Approach Towards Improved Robotic Detection of Marine Litter
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Game of GANs: Game-Theoretical Models for Generative Adversarial Networks
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Multi-Modal Super Resolution for Dense Microscopic Particle Size Estimation
• Generative networks as inverse problems with fractional wavelet scattering networks
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Re-designing cities with conditional adversarial networks
• CSI2Image: Image Reconstruction from Channel State Information Using Generative Adversarial Networks