Improving Diffusion Models for Inverse Problems using Manifold Constraints

Abstract

Recently, diffusion models have been used to solve various inverse problems in an unsupervised manner with appropriate modifications to the sampling process. However, the current solvers, which recursively apply a reverse diffusion step followed by a projection-based measurement consistency step, often produce suboptimal results. By studying the generative sampling path, here we show that current solvers throw the sample path off the data manifold, and hence the error accumulates. To address this, we propose an additional correction term inspired by the manifold constraint, which can be used synergistically with the previous solvers to make the iterations close to the manifold. The proposed manifold constraint is straightforward to implement within a few lines of code, yet boosts the performance by a surprisingly large margin. With extensive experiments, we show that our method is superior to the previous methods both theoretically and empirically, producing promising results in many applications such as image inpainting, colorization, and sparse-view computed tomography. Code available https://github.com/HJ-harry/MCG_diffusion

NeurIPS 2022 camera-ready; 29 pages, 16 figures

Cited by in corpus (11)

• Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Denoising diffusion algorithm for inverse design of microstructures with fine-tuned nonlinear material properties
• A prior regularized full waveform inversion using generative diffusion models
• Conditional score-based diffusion models for solving inverse problems in mechanics
• Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
• Diffusion-Based Audio Inpainting
• Robust multi-coil MRI reconstruction via self-supervised denoising
• Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models
• Application-driven Validation of Posteriors in Inverse Problems
• Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models