Diffusion Model-Based Image Editing: A Survey
arXiv:2402.17525 · doi:10.1109/TPAMI.2025.3541625
Abstract
Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provide an exhaustive overview of existing methods using diffusion models for image editing, covering both theoretical and practical aspects in the field. We delve into a thorough analysis and categorization of these works from multiple perspectives, including learning strategies, user-input conditions, and the array of specific editing tasks that can be accomplished. In addition, we pay special attention to image inpainting and outpainting, and explore both earlier traditional context-driven and current multimodal conditional methods, offering a comprehensive analysis of their methodologies. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval, featuring an innovative metric, LMM Score. Finally, we address current limitations and envision some potential directions for future research. The accompanying repository is released at https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods.
Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2025)
References in corpus (85)
- Conditional Generative Adversarial Nets
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- LoRA: Low-Rank Adaptation of Large Language Models
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Diffusion Models in Vision: A Survey
- Energy-based Generative Adversarial Network
- Blended Diffusion for Text-driven Editing of Natural Images
- Imagen Video: High Definition Video Generation with Diffusion Models
- Blended Latent Diffusion
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
- Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
- SpaText: Spatio-Textual Representation for Controllable Image Generation
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
- Diffusion Models, Image Super-Resolution And Everything: A Survey
- Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
- UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
- EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
- Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model
- MagicVideo: Efficient Video Generation With Latent Diffusion Models
- Inpaint Anything: Segment Anything Meets Image Inpainting
- ModelScope Text-to-Video Technical Report
- Subject-driven Text-to-Image Generation via Apprenticeship Learning
- Diffusion Self-Guidance for Controllable Image Generation
- Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey
- DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
- Efficient Diffusion Models for Vision: A Survey
- MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing
- Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
- Hierarchical Integration Diffusion Model for Realistic Image Deblurring
- Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
- StyleDrop: Text-to-Image Generation in Any Style
- UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
- Inst-Inpaint: Instructing to Remove Objects with Diffusion Models
- VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
- StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing
- Guiding Instruction-based Image Editing via Multimodal Large Language Models
- DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation
- ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation
- High-Resolution Image Editing via Multi-Stage Blended Diffusion
- The Stable Artist: Steering Semantics in Diffusion Latent Space
- InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions
- A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions
- Supervision by Denoising for Medical Image Segmentation
- LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance
- Photoswap: Personalized Subject Swapping in Images
- Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
- Region-Aware Diffusion for Zero-shot Text-driven Image Editing
- A Comprehensive Survey on Generative Diffusion Models for Structured Data
- Fine-grained Image Editing by Pixel-wise Guidance Using Diffusion Models
- EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
- Lasagna: Layered Score Distillation for Disentangled Object Relighting
- Conditional Score Guidance for Text-Driven Image-to-Image Translation
- ChatFace: Chat-Guided Real Face Editing via Diffusion Latent Space Manipulation
- Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models
- DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency
- Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Models
- The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
- CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
- Object-aware Inversion and Reassembly for Image Editing
- ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation
- Neural Gaffer: Relighting Any Object via Diffusion
- Reference-based Image Composition with Sketch via Structure-aware Diffusion Model
- Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models
- DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models
- Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models
- Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
- Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory
- Forgedit: Text Guided Image Editing via Learning and Forgetting
- SUD: Supervision by Denoising Diffusion Models for Image Reconstruction
- WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on
- InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following
- DialogPaint: A Dialog-based Image Editing Model
- Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models
- User-friendly Image Editing with Minimal Text Input: Leveraging Captioning and Injection Techniques
- MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation
- MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers
- PRedItOR: Text Guided Image Editing with Diffusion Prior