Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model

Abstract

Generating and inserting new objects into 3D content is a compelling approach for achieving versatile scene recreation. Existing methods, which rely on SDS optimization or single-view inpainting, often struggle to produce high-quality results. To address this, we propose a novel method for object insertion in 3D content represented by Gaussian Splatting. Our approach introduces a multi-view diffusion model, dubbed MVInpainter, which is built upon a pre-trained stable video diffusion model to facilitate view-consistent object inpainting. Within MVInpainter, we incorporate a ControlNet-based conditional injection module to enable controlled and more predictable multi-view generation. After generating the multi-view inpainted results, we further propose a mask-aware 3D reconstruction technique to refine Gaussian Splatting reconstruction from these sparse inpainted views. By leveraging these fabricate techniques, our approach yields diverse results, ensures view-consistent and harmonious insertions, and produces better object quality. Extensive experiments demonstrate that our approach outperforms existing methods.

arXiv commentAccepted by Visual Informatics. Project Page: https://github.com/JiuTongBro/MultiView_Inpaint

References in corpus (19)

• DreamFusion: Text-to-3D using 2D Diffusion
• NeRF--: Neural Radiance Fields Without Known Camera Parameters
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
• Segment and Track Anything
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
• Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
• Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model
• NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors
• OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields
• ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF
• Text-to-3D with Classifier Score Distillation
• ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
• HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance
• InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
• InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
• ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields
• GO-NeRF: Generating Objects in Neural Radiance Fields for Virtual Reality Content Creation
• Mesh-Guided Neural Implicit Field Editing