ShaRF: Shape-conditioned Radiance Fields from a Single View

Abstract

We present a method for estimating neural scenes representations of objects given only a single image. The core of our method is the estimation of a geometric scaffold for the object and its use as a guide for the reconstruction of the underlying radiance field. Our formulation is based on a generative process that first maps a latent code to a voxelized shape, and then renders it to an image, with the object appearance being controlled by a second latent code. During inference, we optimize both the latent codes and the networks to fit a test image of a new object. The explicit disentanglement of shape and appearance allows our model to be fine-tuned given a single image. We can then render new views in a geometrically consistent manner and they represent faithfully the input object. Additionally, our method is able to generalize to images outside of the training domain (more realistic renderings and even real photographs). Finally, the inferred geometric scaffold is itself an accurate estimate of the object's 3D shape. We demonstrate in several experiments the effectiveness of our approach in both synthetic and real images.

arXiv commentProject page: http://www.krematas.com/sharf/index.html

References in corpus (12)

• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
• MetaSDF: Meta-learning Signed Distance Functions
• PolyGen: An Autoregressive Generative Model of 3D Meshes
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• Neural Face Editing with Intrinsic Image Disentangling
• Neural Volume Rendering: NeRF And Beyond
• IBRNet: Learning Multi-View Image-Based Rendering

Cited by in corpus (11)

• Demystifying the Draft EU Artificial Intelligence Act
• Fast Learning Radiance Fields by Shooting Much Fewer Rays
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
• Unsupervised Discovery of Object Radiance Fields
• SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
• DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering
• BARF: Bundle-Adjusting Neural Radiance Fields
• Neural Rays for Occlusion-aware Image-based Rendering
• Learning to Stylize Novel Views
• Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
• Fast and Explicit Neural View Synthesis