SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Abstract

In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.

arXiv commentBMVC 2018 camera ready

References in corpus (12)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Training of Wasserstein GANs
• Improved Techniques for Training GANs
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Deep Image Prior
• Least Squares Generative Adversarial Networks
• Generative Face Completion
• Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart
• Contextual-based Image Inpainting: Infer, Match, and Translate

Cited by in corpus (21)

• AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
• Adversarial Inpainting of Medical Image Modalities
• StructureFlow: Image Inpainting via Structure-aware Appearance Flow
• Efficient texture-aware multi-GAN for image inpainting
• Facial-Sketch Synthesis: A New Challenge
• SPGNet: Semantic Prediction Guidance for Scene Parsing
• Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations
• Image inpainting using frequency domain priors
• Deep Face Video Inpainting via UV Mapping
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes
• PI-REC: Progressive Image Reconstruction Network With Edge and Color Domain
• Image Inpainting Guided by Coherence Priors of Semantics and Textures
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Bridging Global Context Interactions for High-Fidelity Image Completion
• TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations
• Learning Oracle Attention for High-fidelity Face Completion
• Deep Fusion Network for Image Completion
• Semantic Editing On Segmentation Map Via Multi-Expansion Loss
• One-Stage Inpainting with Bilateral Attention and Pyramid Filling Block
• Synthesizing Photorealistic Images with Deep Generative Learning
• Environment reconstruction on depth images using Generative Adversarial Networks