Generative Image Inpainting with Contextual Attention

Abstract

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting.

arXiv commentAccepted in CVPR 2018; add CelebA-HQ results; open sourced; interactive demo available: http://jhyu.me/demo

References in corpus (13)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Training of Wasserstein GANs
• Improved Techniques for Training GANs
• Deformable Convolutional Networks
• Video Inpainting of Complex Scenes
• Fast Patch-based Style Transfer of Arbitrary Style
• Least Squares Generative Adversarial Networks
• Generative Face Completion
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
• Active Convolution: Learning the Shape of Convolution for Image Classification

Cited by in corpus (67)

• Graph Self-Supervised Learning: A Survey
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Medical Deep Learning -- A systematic Meta-Review
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• Image De-raining Using a Conditional Generative Adversarial Network
• A deep learning framework for quality assessment and restoration in video endoscopy
• Deep Feature Augmentation for Occluded Image Classification
• Learnable Gated Temporal Shift Module for Deep Video Inpainting
• MTRNet++: One-stage Mask-based Scene Text Eraser
• Robust Conditional Generative Adversarial Networks
• DeepStreet: A deep learning powered urban street network generation module
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Image Inpainting for Irregular Holes Using Partial Convolutions
• Image Inpainting via Conditional Texture and Structure Dual Generation
• 3D Photography using Context-aware Layered Depth Inpainting
• A Pixel-Based Framework for Data-Driven Clothing
• Examining Autocompletion as a Basic Concept for Interaction with Generative AI
• Human Motion Prediction via Spatio-Temporal Inpainting
• Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
• PISE: Person Image Synthesis and Editing with Decoupled GAN
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• Where to Look Next: Unsupervised Active Visual Exploration on 360° Input
• VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
• Structured Output Learning with Conditional Generative Flows
• Chest X-ray Inpainting with Deep Generative Models
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning
• In-Distribution Interpretability for Challenging Modalities
• Planning Paths Through Unknown Space by Imagining What Lies Therein
• Attentive Normalization for Conditional Image Generation
• Deep Residual Mixture Models
• Artist Style Transfer Via Quadratic Potential
• Hallucinating very low-resolution and obscured face images
• Deep Flow-Guided Video Inpainting
• Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools
• Boosting Image Outpainting with Semantic Layout Prediction
• Context-Aware Image Inpainting with Learned Semantic Priors
• PD-GAN: Probabilistic Diverse GAN for Image Inpainting
• What and Where: A Context-based Recommendation System for Object Insertion
• Learning Symmetry Consistent Deep CNNs for Face Completion
• Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence
• Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement
• Texture Transform Attention for Realistic Image Inpainting
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• DVI: Depth Guided Video Inpainting for Autonomous Driving
• Generator Versus Segmentor: Pseudo-healthy Synthesis
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
• Guided Image Inpainting: Replacing an Image Region by Pulling Content from Another Image
• One-Stage Inpainting with Bilateral Attention and Pyramid Filling Block
• Future Video Synthesis with Object Motion Prediction
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Automatic Segmentation of Non-Tumor Tissues in Glioma MR Brain Images Using Deformable Registration with Partial Convolutional Networks
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos
• Pixel-wise Conditioning of Generative Adversarial Networks
• Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance
• Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation
• Align-and-Attend Network for Globally and Locally Coherent Video Inpainting
• Unconstrained Foreground Object Search
• ByeGlassesGAN: Identity Preserving Eyeglasses Removal for Face Images
• Synthesizing Photorealistic Images with Deep Generative Learning
• Context Encoding Chest X-rays
• Sparse to Dense Motion Transfer for Face Image Animation
• Generative Imaging and Image Processing via Generative Encoder