Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process

Abstract

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.

This paper has been published on IEEE Transactions on Image Processing

References in corpus (3)

• Massive MIMO for Next Generation Wireless Systems
• Scaling up MIMO: Opportunities and Challenges with Very Large Arrays
• Overview of Millimeter Wave Communications for Fifth-Generation (5G) Wireless Networks-with a focus on Propagation Models

Cited by in corpus (5)

• Joint Visibility Region and Channel Estimation for Extremely Large-scale MIMO Systems
• Revisiting XL-MIMO Channel Estimation: When Dual-Wideband Effects Meet Near Field
• Recent Activities of a European Union Joint Research Project on Metrology for Emerging Wireless Standards
• Low-Complexity On-Grid Channel Estimation for Partially-Connected Hybrid XL-MIMO
• Performance Analysis of STAR-RIS-Assisted NOMA Wireless Systems with Realistic Indoor Outdoor THz Channel Models