Image Processing Using Multi-Code GAN Prior

Abstract

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, applying trained GAN models to real image processing remains challenging. Previous methods typically invert a target image back to the latent space either by back-propagation or by learning an additional encoder. However, the reconstructions from both of the methods are far from ideal. In this work, we propose a novel approach, called mGANprior, to incorporate the well-trained GANs as effective prior to a variety of image processing tasks. In particular, we employ multiple latent codes to generate multiple feature maps at some intermediate layer of the generator, then compose them with adaptive channel importance to recover the input image. Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing competitors. The resulting high-fidelity image reconstruction enables the trained GAN models as prior to many real-world applications, such as image colorization, super-resolution, image inpainting, and semantic manipulation. We further analyze the properties of the layer-wise representation learned by GAN models and shed light on what knowledge each layer is capable of representing.

CVPR2020 camera-ready

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Invertible Conditional GANs for image editing
• Fader Networks: Manipulating Images by Sliding Attributes
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
• Latent Convolutional Models

Cited by in corpus (2)

• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Wavelet Transform-assisted Adaptive Generative Modeling for Colorization