Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

Abstract

Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as corrupted image regions may have arbitrary shapes with contexts from any direction. We propose BAT-Fill, an innovative image inpainting framework that introduces a novel bidirectional autoregressive transformer (BAT) for image inpainting. BAT utilizes the transformers to learn autoregressive distributions, which naturally allows the diverse generation of missing contents. In addition, it incorporates the masked language model like BERT, which enables bidirectionally modeling of contextual information of missing regions for better image completion. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.

arXiv comment11 pages, 6 figures

References in corpus (11)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• Towards Deeper Understanding of Variational Autoencoding Models
• EMLight: Lighting Estimation via Spherical Distribution Approximation
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE

Cited by in corpus (4)

• Blind Image Super-Resolution via Contrastive Representation Learning
• WaveFill: A Wavelet-based Generation Network for Image Inpainting
• FBC-GAN: Diverse and Flexible Image Synthesis via Foreground-Background Composition
• Bi-level Feature Alignment for Versatile Image Translation and Manipulation