DiffEdit: Diffusion-based semantic image editing with mask guidance
arXiv:2210.11427
Abstract
Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. DiffEdit achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images.
arXiv commentPreprint
References in corpus (17)
- Denoising Diffusion Probabilistic Models
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Diffusion Models Beat GANs on Image Synthesis
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Zero-Shot Text-to-Image Generation
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- Blended Diffusion for Text-driven Editing of Natural Images
- Classifier-Free Diffusion Guidance
- CogView: Mastering Text-to-Image Generation via Transformers
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
- Prompt-to-Prompt Image Editing with Cross Attention Control
- SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
- Pretraining is All You Need for Image-to-Image Translation
- Image-to-Image Translation with Text Guidance
- Understanding DDPM Latent Codes Through Optimal Transport
- Paint by Word
- A Benchmark and Baseline for Language-Driven Image Editing
Cited by in corpus (7)
- SpaText: Spatio-Textual Representation for Controllable Image Generation
- PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
- Synthetic Data Generation Using Large Language Models: Advances in Text and Code
- LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance
- Face Aging via Diffusion-based Editing
- Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations
- Fidelity-preserving enhancement of ptychography with foundational text-to-image models