Toward Interactive Modulation for Photo-Realistic Image Restoration

Abstract

Modulating image restoration level aims to generate a restored image by altering a factor that represents the restoration strength. Previous works mainly focused on optimizing the mean squared reconstruction error, which brings high reconstruction accuracy but lacks finer texture details. This paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate high-frequency textures in the modulation tasks. CUGAN consists of two modules -- base networks and condition networks. The base networks comprise a generator and a discriminator. In the generator, we realize the interactive control of restoration levels by tuning the weights of different features from different scales in the Unet architecture. Moreover, we adaptively modulate the intermediate features in the discriminator according to the severity of degradations. The condition networks accept the condition vector (encoded degradation information) as input, then generate modulation parameters for both the generator and the discriminator. During testing, users can control the output effects by tweaking the condition vector. We also provide a smooth transition between GAN and MSE effects by a simple transition method. Extensive experiments demonstrate that the proposed CUGAN achieves excellent performance on image restoration modulation tasks.

References in corpus (7)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Image Quality Assessment: Unifying Structure and Texture Similarity
• Density estimation using Real NVP
• Natural Image Noise Dataset
• Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration