Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning

Abstract

This paper presents a new adversarial training framework for image inpainting with segmentation confusion adversarial training (SCAT) and contrastive learning. SCAT plays an adversarial game between an inpainting generator and a segmentation network, which provides pixel-level local training signals and can adapt to images with free-form holes. By combining SCAT with standard global adversarial training, the new adversarial training framework exhibits the following three advantages simultaneously: (1) the global consistency of the repaired image, (2) the local fine texture details of the repaired image, and (3) the flexibility of handling images with free-form holes. Moreover, we propose the textural and semantic contrastive learning losses to stabilize and improve our inpainting model's training by exploiting the feature representation space of the discriminator, in which the inpainting images are pulled closer to the ground truth images but pushed farther from the corrupted images. The proposed contrastive losses better guide the repaired images to move from the corrupted image data points to the real image data points in the feature representation space, resulting in more realistic completed images. We conduct extensive experiments on two benchmark datasets, demonstrating our model's effectiveness and superiority both qualitatively and quantitatively.

Accepted to AAAI2023, Oral

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Simple Framework for Contrastive Learning of Visual Representations
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Momentum Contrast for Unsupervised Visual Representation Learning
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• Contrastive Learning for Unpaired Image-to-Image Translation
• ContraGAN: Contrastive Learning for Conditional Image Generation
• Geometric GAN
• MAT: Mask-Aware Transformer for Large Hole Image Inpainting
• Image Inpainting via Conditional Texture and Structure Dual Generation
• Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding
• Reduce Information Loss in Transformers for Pluralistic Image Inpainting
• WaveFill: A Wavelet-based Generation Network for Image Inpainting
• High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling