Passive Non-Line-of-Sight Imaging with Light Transport Modulation

Abstract

Passive non-line-of-sight (NLOS) imaging has witnessed rapid development in recent years, due to its ability to image objects that are out of sight. The light transport condition plays an important role in this task since changing the conditions will lead to different imaging models. Existing learning-based NLOS methods usually train independent models for different light transport conditions, which is computationally inefficient and impairs the practicality of the models. In this work, we propose NLOS-LTM, a novel passive NLOS imaging method that effectively handles multiple light transport conditions with a single network. We achieve this by inferring a latent light transport representation from the projection image and using this representation to modulate the network that reconstructs the hidden image from the projection image. We train a light transport encoder together with a vector quantizer to obtain the light transport representation. To further regulate this representation, we jointly learn both the reconstruction network and the reprojection network during training. A set of light transport modulation blocks is used to modulate the two jointly trained networks in a multi-scale way. Extensive experiments on a large-scale passive NLOS dataset demonstrate the superiority of the proposed method. The code is available at https://github.com/JerryOctopus/NLOS-LTM.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Representation Learning with Contrastive Predictive Coding
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Non-invasive real-time imaging through scattering layers and around corners via speckle correlations
• Deep Dense Multi-scale Network for Snow Removal Using Semantic and Geometric Priors
• Denoising Diffusion Restoration Models
• Non-line-of-sight imaging with arbitrary illumination and detection pattern
• Lensless Photography with only an image sensor
• KBNet: Kernel Basis Network for Image Restoration
• Flash Photography for Data-Driven Hidden Scene Recovery
• Non-line-of-sight imaging off a Phong surface through deep learning
• Occlusion Fields: An Implicit Representation for Non-Line-of-Sight Surface Reconstruction