FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models

Abstract

Recent works on generalizable NeRFs have shown promising results on novel view synthesis from single or few images. However, such models have rarely been applied on other downstream tasks beyond synthesis such as semantic understanding and parsing. In this paper, we propose a novel framework named FeatureNeRF to learn generalizable NeRFs by distilling pre-trained vision foundation models (e.g., DINO, Latent Diffusion). FeatureNeRF leverages 2D pre-trained foundation models to 3D space via neural rendering, and then extract deep features for 3D query points from NeRF MLPs. Consequently, it allows to map 2D images to continuous 3D semantic feature volumes, which can be used for various downstream tasks. We evaluate FeatureNeRF on tasks of 2D/3D semantic keypoint transfer and 2D/3D object part segmentation. Our extensive experiments demonstrate the effectiveness of FeatureNeRF as a generalizable 3D semantic feature extractor. Our project page is available at https://jianglongye.com/featurenerf/ .

arXiv commentProject page: https://jianglongye.com/featurenerf/

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Denoising Diffusion Probabilistic Models
• ShapeNet: An Information-Rich 3D Model Repository
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• On the Opportunities and Risks of Foundation Models
• Diffusion Models Beat GANs on Image Synthesis
• Alias-Free Generative Adversarial Networks
• Florence: A New Foundation Model for Computer Vision
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Self-Supervised Learning with Swin Transformers
• Decomposing NeRF for Editing via Feature Field Distillation
• Diffusion Models for Implicit Image Segmentation Ensembles
• SPair-71k: A Large-scale Benchmark for Semantic Correspondence
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
• Learning 3D Dense Correspondence via Canonical Point Autoencoder