DriftRec: Adapting diffusion models to blind JPEG restoration

Abstract

In this work, we utilize the high-fidelity generation abilities of diffusion models to solve blind JPEG restoration at high compression levels. We propose an elegant modification of the forward stochastic differential equation of diffusion models to adapt them to this restoration task and name our method DriftRec. Comparing DriftRec against an $L_2$ regression baseline with the same network architecture and state-of-the-art techniques for JPEG restoration, we show that our approach can escape the tendency of other methods to generate blurry images, and recovers the distribution of clean images significantly more faithfully. For this, only a dataset of clean/corrupted image pairs and no knowledge about the corruption operation is required, enabling wider applicability to other restoration tasks. In contrast to other conditional and unconditional diffusion models, we utilize the idea that the distributions of clean and corrupted images are much closer to each other than each is to the usual Gaussian prior of the reverse process in diffusion models. Our approach therefore requires only low levels of added noise and needs comparatively few sampling steps even without further optimizations. We show that DriftRec naturally generalizes to realistic and difficult scenarios such as unaligned double JPEG compression and blind restoration of JPEGs found online, without having encountered such examples during training.

(C) 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

References in corpus (10)

• scikit-image: Image processing in Python
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• The Perception-Distortion Tradeoff
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
• Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration
• Soft Diffusion: Score Matching for General Corruptions
• Modify the Improved Euler scheme to integrate stochastic differential equations
• DiffPhase: Generative Diffusion-based STFT Phase Retrieval