Painterly Image Harmonization using Diffusion Model

Abstract

Painterly image harmonization aims to insert photographic objects into paintings and obtain artistically coherent composite images. Previous methods for this task mainly rely on inference optimization or generative adversarial network, but they are either very time-consuming or struggling at fine control of the foreground objects (e.g., texture and content details). To address these issues, we propose a novel Painterly Harmonization stable Diffusion model (PHDiffusion), which includes a lightweight adaptive encoder and a Dual Encoder Fusion (DEF) module. Specifically, the adaptive encoder and the DEF module first stylize foreground features within each encoder. Then, the stylized foreground features from both encoders are combined to guide the harmonization process. During training, besides the noise loss in diffusion model, we additionally employ content loss and two style losses, i.e., AdaIN style loss and contrastive style loss, aiming to balance the trade-off between style migration and content preservation. Compared with the state-of-the-art models from related fields, our PHDiffusion can stylize the foreground more sufficiently and simultaneously retain finer content. Our code and model are available at https://github.com/bcmi/PHDiffusion-Painterly-Image-Harmonization.

arXiv commentAccepted by ACMMM 2023

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Denoising Diffusion Probabilistic Models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Universal Style Transfer via Feature Transforms
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
• Adding Conditional Control to Text-to-Image Diffusion Models
• Improving the Harmony of the Composite Image by Spatial-Separated Attention Module
• Pseudo Numerical Methods for Diffusion Models on Manifolds
• GP-GAN: Towards Realistic High-Resolution Image Blending
• Diffusion-based Image Translation using Disentangled Style and Content Representation
• Making Images Real Again: A Comprehensive Survey on Deep Image Composition
• StyTr${}^2$: Image Style Transfer with Transformers
• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
• Cross-domain Compositing with Pretrained Diffusion Models
• Training-free Content Injection using h-space in Diffusion Models
• Inversion-Based Style Transfer with Diffusion Models