DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems

Abstract

Image-based relighting, projector compensation and depth/normal reconstruction are three important tasks of projector-camera systems (ProCams) and spatial augmented reality (SAR). Although they share a similar pipeline of finding projector-camera image mappings, in tradition, they are addressed independently, sometimes with different prerequisites, devices and sampling images. In practice, this may be cumbersome for SAR applications to address them one-by-one. In this paper, we propose a novel end-to-end trainable model named DeProCams to explicitly learn the photometric and geometric mappings of ProCams, and once trained, DeProCams can be applied simultaneously to the three tasks. DeProCams explicitly decomposes the projector-camera image mappings into three subprocesses: shading attributes estimation, rough direct light estimation and photorealistic neural rendering. A particular challenge addressed by DeProCams is occlusion, for which we exploit epipolar constraint and propose a novel differentiable projector direct light mask. Thus, it can be learned end-to-end along with the other modules. Afterwards, to improve convergence, we apply photometric and geometric constraints such that the intermediate results are plausible. In our experiments, DeProCams shows clear advantages over previous arts with promising quality and meanwhile being fully differentiable. Moreover, by solving the three tasks in a unified model, DeProCams waives the need for additional optical devices, radiometric calibrations and structured light.

arXiv commentSource code and supplementary material at: https://github.com/BingyaoHuang/DeProCams

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• Neural Inverse Rendering for General Reflectance Photometric Stereo
• End-to-end Projector Photometric Compensation
• End-to-end Full Projector Compensation
• CompenNet++: End-to-end Full Projector Compensation
• IlluminatedFocus: Vision Augmentation using Spatial Defocusing via Focal Sweep Eyeglasses and High-Speed Projector
• Pro-Cam SSfM: Projector-Camera System for Structure and Spectral Reflectance from Motion

Cited by in corpus (5)

• End-to-end Full Projector Compensation
• LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization
• DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
• GS-ProCams: Gaussian Splatting-based Projector-Camera Systems
• ProCap: Projection-Aware Captioning for Spatial Augmented Reality