Joint multi-dimensional dynamic attention and transformer for general image restoration

Abstract

Outdoor images often suffer from severe degradation due to rain, haze, and noise, impairing image quality and challenging high-level tasks. Current image restoration methods struggle to handle complex degradation while maintaining efficiency. This paper introduces a novel image restoration architecture that combines multi-dimensional dynamic attention and self-attention within a U-Net framework. To leverage the global modeling capabilities of transformers and the local modeling capabilities of convolutions, we integrate sole CNNs in the encoder-decoder and sole transformers in the latent layer. Additionally, we design convolutional kernels with selected multi-dimensional dynamic attention to capture diverse degraded inputs efficiently. A transformer block with transposed self-attention further enhances global feature extraction while maintaining efficiency. Extensive experiments demonstrate that our method achieves a better balance between performance and computational complexity across five image restoration tasks: deraining, deblurring, denoising, dehazing, and enhancement, as well as superior performance for high-level vision tasks. The source code will be available at https://github.com/House-yuyu/MDDA-former.

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• DehazeNet: An End-to-End System for Single Image Haze Removal
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising
• Vision Transformers for Single Image Dehazing
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Clearing the Skies: A deep network architecture for single-image rain removal
• BANet: Blur-aware Attention Networks for Dynamic Scene Deblurring
• Dual-former: Hybrid Self-attention Transformer for Efficient Image Restoration