DreamFusion: Text-to-3D using 2D Diffusion

Abstract

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

arXiv commentsee project page at https://dreamfusion3d.github.io/

Cited by in corpus (19)

• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
• The Infinite Index: Information Retrieval on Generative Text-To-Image Models
• Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases
• Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image
• LGTM: Local-to-Global Text-Driven Human Motion Diffusion Model
• CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality
• A Survey on Personalized Content Synthesis with Diffusion Models
• SlimmeRF: Slimmable Radiance Fields
• Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly
• Text2Avatar: Text to 3D Human Avatar Generation with Codebook-Driven Body Controllable Attribute
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
• Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model
• Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations
• ReShader: View-Dependent Highlights for Single Image View-Synthesis
• MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields
• CharNeRF: 3D Character Generation from Concept Art
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting