Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images

Abstract

We present neural radiance fields (NeRF) with templates, dubbed Template-NeRF, for modeling appearance and geometry and generating dense shape correspondences simultaneously among objects of the same category from only multi-view posed images, without the need of either 3D supervision or ground-truth correspondence knowledge. The learned dense correspondences can be readily used for various image-based tasks such as keypoint detection, part segmentation, and texture transfer that previously require specific model designs. Our method can also accommodate annotation transfer in a one or few-shot manner, given only one or a few instances of the category. Using periodic activation and feature-wise linear modulation (FiLM) conditioning, we introduce deep implicit templates on 3D data into the 3D-aware image synthesis pipeline NeRF. By representing object instances within the same category as shape and appearance variation of a shared NeRF template, our proposed method can achieve dense shape correspondences reasoning on images for a wide range of object classes. We demonstrate the results and applications on both synthetic and real-world data with competitive results compared with other methods based on 3D information.

arXiv comment10 pages, 8 figures

References in corpus (10)

• NeRF++: Analyzing and Improving Neural Radiance Fields
• Universal Correspondence Network
• Self-supervised learning of a facial attribute embedding from video
• Neural Reflectance Fields for Appearance Acquisition
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• CR-GAN: Learning Complete Representations for Multi-view Generation
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
• Semantic Correspondence via 2D-3D-2D Cycle
• Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence
• Deep Implicit Templates for 3D Shape Representation