Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Abstract

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

ICLR 2021 (Spotlight). Code: https://github.com/zsyzzsoft/co-mod-gan Demo: https://comodgan.ml/

References in corpus (13)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• The Perception-Distortion Tradeoff
• A Learned Representation For Artistic Style
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• Assessing Generative Models via Precision and Recall
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Is Generator Conditioning Causally Related to GAN Performance?
• On the Evaluation of Conditional GANs
• Painting Outside the Box: Image Outpainting with GANs
• LaFIn: Generative Landmark Guided Face Inpainting
• Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network
• Stochastic Conditional Generative Networks with Basis Decomposition

Cited by in corpus (12)

• A Systematic Survey of Regularization and Normalization in GANs
• Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning
• 360-Degree Panorama Generation from Few Unregistered NFoV Images
• Guided Co-Modulated GAN for 360° Field of View Extrapolation
• Deep Face Video Inpainting via UV Mapping
• Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
• PEEL: A Provable Removal Attack on Deep Hiding
• A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
• Generative Adversarial Networks in Ultrasound Imaging: Extending Field of View Beyond Conventional Limits
• The Image Local Autoregressive Transformer
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
• EdiBERT, a generative model for image editing