Zero-Shot Text-Guided Object Generation with Dream Fields

Abstract

We combine neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions. Our method, Dream Fields, can generate the geometry and color of a wide range of objects without 3D supervision. Due to the scarcity of diverse, captioned 3D data, prior methods only generate objects from a handful of categories, such as ShapeNet. Instead, we guide generation with image-text models pre-trained on large datasets of captioned images from the web. Our method optimizes a Neural Radiance Field from many camera views so that rendered images score highly with a target caption according to a pre-trained CLIP model. To improve fidelity and visual quality, we introduce simple geometric priors, including sparsity-inducing transmittance regularization, scene bounds, and new MLP architectures. In experiments, Dream Fields produce realistic, multi-view consistent object geometry and color from a variety of natural language captions.

arXiv commentCVPR 2022. 13 pages. Website: https://ajayj.com/dreamfields

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• Diffusion Models Beat GANs on Image Synthesis
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Zero-Shot Text-to-Image Generation
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• NeRF++: Analyzing and Improving Neural Radiance Fields
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
• Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
• Learning to Compose Visual Relations
• Learning Gradient Fields for Shape Generation
• Evolving Evocative 2D Views of Generated 3D Objects