MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers

Abstract

Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: https://github.com/taco-group/MWFormer

Accepted by IEEE Transactions on Image Processing. The code is available at: https://github.com/taco-group/MWFormer

References in corpus (8)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• A Simple Framework for Contrastive Learning of Visual Representations
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• DesnowNet: Context-Aware Deep Network for Snow Removal
• Deep Dense Multi-scale Network for Snow Removal Using Semantic and Geometric Priors
• LMQFormer: A Laplace-Prior-Guided Mask Query Transformer for Lightweight Snow Removal
• GIA-Net: Global Information Aware Network for Low-light Imaging

Cited by in corpus (1)

• A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends