MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion

Abstract

Reconstructing 3D models from single-view images is a long-standing problem in computer vision. The latest advances for single-image 3D reconstruction extract a textual description from the input image and further utilize it to synthesize 3D models. However, existing methods focus on capturing a single key attribute of the image (e.g., object type, artistic style) and fail to consider the multi-perspective information required for accurate 3D reconstruction, such as object shape and material properties. Besides, the reliance on Neural Radiance Fields hinders their ability to reconstruct intricate surfaces and texture details. In this work, we propose MTFusion, which leverages both image data and textual descriptions for high-fidelity 3D reconstruction. Our approach consists of two stages. First, we adopt a novel multi-word textual inversion technique to extract a detailed text description capturing the image's characteristics. Then, we use this description and the image to generate a 3D model with FlexiCubes. Additionally, MTFusion enhances FlexiCubes by employing a special decoder network for Signed Distance Functions, leading to faster training and finer surface representation. Extensive evaluations demonstrate that our MTFusion surpasses existing image-to-3D methods on a wide range of synthetic and real-world images. Furthermore, the ablation study proves the effectiveness of our network designs.

PRCV 2024

References in corpus (9)

• Learning Transferable Visual Models From Natural Language Supervision
• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
• U${}^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection
• LAION-5B: An open large-scale dataset for training next generation image-text models
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
• Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
• Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors