Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Abstract

We present a method for generating colored 3D shapes from natural language. To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes. Our model combines and extends learning by association and metric learning approaches to learn implicit cross-modal connections, and produces a joint representation that captures the many-to-many relations between language and physical properties of 3D shapes such as color and shape. To evaluate our approach, we collect a large dataset of natural language descriptions for physical 3D objects in the ShapeNet dataset. With this learned joint embedding we demonstrate text-to-shape retrieval that outperforms baseline approaches. Using our embeddings with a novel conditional Wasserstein GAN framework, we generate colored 3D shapes from text. Our method is the first to connect natural language text with realistic 3D objects exhibiting rich variations in color, texture, and shape detail. See video at https://youtu.be/zraPvRdl13Q

References in corpus (13)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Improved Training of Wasserstein GANs
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• Learning by Association - A versatile semi-supervised training method for neural networks
• Associative Domain Adaptation
• Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis
• Learning Robust Visual-Semantic Embeddings
• Interactive 3D Modeling with a Generative Adversarial Network

Cited by in corpus (10)

• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• Zero-Shot Text-Guided Object Generation with Dream Fields
• Learning Material-Aware Local Descriptors for 3D Shapes
• Learning a Disentangled Embedding for Monocular 3D Shape Retrieval and Pose Estimation
• Generation High resolution 3D model from natural language by Generative Adversarial Network
• Be Consistent! Improving Procedural Text Comprehension using Label Consistency
• Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences
• ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences
• Let's Transfer Transformations of Shared Semantic Representations
• Static and Animated 3D Scene Generation from Free-form Text Descriptions