Semantic Image Inpainting with Deep Generative Models

Abstract

Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.

Cited by in corpus (16)

• Towards a Robust Parameterization for Conditioning Facies Models Using Deep Variational Autoencoders and Ensemble Smoother
• What Do We Understand About Convolutional Networks?
• Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search
• DeepFlow: History Matching in the Space of Deep Generative Models
• Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data
• Semantic Image Completion and Enhancement using Deep Learning
• Contextual-based Image Inpainting: Infer, Match, and Translate
• Learning Disentangling and Fusing Networks for Face Completion Under Structured Occlusions
• The Reincarnation of Grille Cipher: A Generative Approach
• ChaLearn Looking at People: Inpainting and Denoising challenges
• Extreme 3D Face Reconstruction: Seeing Through Occlusions
• Training Generative Adversarial Networks in One Stage
• Learning Symmetry Consistent Deep CNNs for Face Completion
• Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
• Generative Model for Heterogeneous Inference
• Synthesizing Photorealistic Images with Deep Generative Learning