Specular-to-Diffuse Translation for Multi-View Reconstruction

Abstract

Most multi-view 3D reconstruction algorithms, especially when shape-from-shading cues are used, assume that object appearance is predominantly diffuse. To alleviate this restriction, we introduce S2Dnet, a generative adversarial network for transferring multiple views of objects with specular reflection into diffuse ones, so that multi-view reconstruction methods can be applied more effectively. Our network extends unsupervised image-to-image translation to multi-view "specular to diffuse" translation. To preserve object appearance across multiple views, we introduce a Multi-View Coherence loss (MVC) that evaluates the similarity and faithfulness of local patches after the view-transformation. Our MVC loss ensures that the similarity of local correspondences among multi-view images is preserved under the image-to-image translation. As a result, our network yields significantly better results than several single-view baseline techniques. In addition, we carefully design and generate a large synthetic training data set using physically-based rendering. During testing, our network takes only the raw glossy images as input, without extra information such as segmentation masks or lighting estimation. Results demonstrate that multi-view reconstruction can be significantly improved using the images filtered by our network. We also show promising performance on real world training and testing data.

arXiv commentAccepted to ECCV 2018

References in corpus (14)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Perceptual Adversarial Networks for Image-to-Image Transformation
• Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
• LIFT: Learned Invariant Feature Transform
• One-Sided Unsupervised Domain Mapping
• Attentive Generative Adversarial Network for Raindrop Removal from a Single Image
• Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal
• TextureGAN: Controlling Deep Image Synthesis with Texture Patches
• How useful is photo-realistic rendering for visual learning?
• Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation: the Benefit of Target Expectation Maximization
• LIME: Live Intrinsic Material Estimation
• Semantic Photometric Bundle Adjustment on Natural Sequences
• Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
• Taming Adversarial Domain Transfer with Structural Constraints for Image Enhancement