High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis

Abstract

Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal. While these learning-based methods are significantly more effective in capturing high-level features than prior techniques, they can only handle very low-resolution inputs due to memory limitations and difficulty in training. Even for slightly larger images, the inpainted regions would appear blurry and unpleasant boundaries become visible. We propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints, which not only preserves contextual structures but also produces high-frequency details by matching and adapting patches with the most similar mid-layer feature correlations of a deep classification network. We evaluate our method on the ImageNet and Paris Streetview datasets and achieved state-of-the-art inpainting accuracy. We show our approach produces sharper and more coherent results than prior methods, especially for high-resolution images.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• Going Deeper with Convolutions
• DRAW: A Recurrent Neural Network For Image Generation
• Fully Convolutional Networks for Semantic Segmentation
• Learning Deconvolution Network for Semantic Segmentation
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Loss Functions for Neural Networks for Image Processing
• Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Towards Automatic Image Editing: Learning to See another You

Cited by in corpus (14)

• HDR image reconstruction from a single exposure using deep CNNs
• Generative Image Inpainting with Contextual Attention
• Structural inpainting
• Visual Attribute Transfer through Deep Image Analogy
• Improving Video Generation for Multi-functional Applications
• Painting baryons onto N-body simulations of galaxy clusters with image-to-image deep learning
• Learning Disentangling and Fusing Networks for Face Completion Under Structured Occlusions
• Inpainting hydrodynamical maps with deep learning
• Context-Aware Semantic Inpainting
• Deep Stacked Networks with Residual Polishing for Image Inpainting
• Extreme 3D Face Reconstruction: Seeing Through Occlusions
• Multiple Sclerosis Lesion Inpainting Using Non-Local Partial Convolutions
• Image Inpainting for High-Resolution Textures using CNN Texture Synthesis
• An Edge Information and Mask Shrinking Based Image Inpainting Approach