Generative Image Inpainting with Contextual Attention

Abstract

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting.

Accepted in CVPR 2018; add CelebA-HQ results; open sourced; interactive demo available: http://jhyu.me/demo

References in corpus (4)

• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Fast Patch-based Style Transfer of Arbitrary Style
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• Active Convolution: Learning the Shape of Convolution for Image Classification

Cited by in corpus (66)

• Graph Self-Supervised Learning: A Survey
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Medical Deep Learning -- A systematic Meta-Review
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• A deep learning framework for quality assessment and restoration in video endoscopy
• Deep Feature Augmentation for Occluded Image Classification
• Learnable Gated Temporal Shift Module for Deep Video Inpainting
• MTRNet++: One-stage Mask-based Scene Text Eraser
• Robust Conditional Generative Adversarial Networks
• DeepStreet: A deep learning powered urban street network generation module
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Image Inpainting for Irregular Holes Using Partial Convolutions
• 3D Photography using Context-aware Layered Depth Inpainting
• Image Inpainting via Conditional Texture and Structure Dual Generation
• A Pixel-Based Framework for Data-Driven Clothing
• Examining Autocompletion as a Basic Concept for Interaction with Generative AI
• Human Motion Prediction via Spatio-Temporal Inpainting
• Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
• PISE: Person Image Synthesis and Editing with Decoupled GAN
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
• Structured Output Learning with Conditional Generative Flows
• Where to Look Next: Unsupervised Active Visual Exploration on 360° Input
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Chest X-ray Inpainting with Deep Generative Models
• Attentive Normalization for Conditional Image Generation
• Deep Residual Mixture Models
• Planning Paths Through Unknown Space by Imagining What Lies Therein
• In-Distribution Interpretability for Challenging Modalities
• Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools
• Artist Style Transfer Via Quadratic Potential
• Hallucinating very low-resolution and obscured face images
• Deep Flow-Guided Video Inpainting
• Context-Aware Image Inpainting with Learned Semantic Priors
• Boosting Image Outpainting with Semantic Layout Prediction
• PD-GAN: Probabilistic Diverse GAN for Image Inpainting
• What and Where: A Context-based Recommendation System for Object Insertion
• Learning Symmetry Consistent Deep CNNs for Face Completion
• Texture Transform Attention for Realistic Image Inpainting
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• Generator Versus Segmentor: Pseudo-healthy Synthesis
• DVI: Depth Guided Video Inpainting for Autonomous Driving
• Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
• Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence
• Guided Image Inpainting: Replacing an Image Region by Pulling Content from Another Image
• Automatic Segmentation of Non-Tumor Tissues in Glioma MR Brain Images Using Deformable Registration with Partial Convolutional Networks
• One-Stage Inpainting with Bilateral Attention and Pyramid Filling Block
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• Future Video Synthesis with Object Motion Prediction
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Align-and-Attend Network for Globally and Locally Coherent Video Inpainting
• Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation
• Sparse to Dense Motion Transfer for Face Image Animation
• Generative Imaging and Image Processing via Generative Encoder
• ByeGlassesGAN: Identity Preserving Eyeglasses Removal for Face Images
• Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance
• Context Encoding Chest X-rays
• Unconstrained Foreground Object Search
• A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos
• Pixel-wise Conditioning of Generative Adversarial Networks
• Synthesizing Photorealistic Images with Deep Generative Learning