Interactive 3D Modeling with a Generative Adversarial Network

Abstract

This paper proposes the idea of using a generative adversarial network (GAN) to assist a novice user in designing real-world shapes with a simple interface. The user edits a voxel grid with a painting interface (like Minecraft). Yet, at any time, he/she can execute a SNAP command, which projects the current voxel grid onto a latent shape manifold with a learned projection operator and then generates a similar, but more realistic, shape using a learned generator network. Then the user can edit the resulting shape and snap again until he/she is satisfied with the result. The main advantage of this approach is that the projection and generation operators assist novice users to create 3D models characteristic of a background distribution of object shapes, but without having to specify all the details. The core new research idea is to use a GAN to support this application. 3D GANs have previously been used for shape generation, interpolation, and completion, but never for interactive modeling. The new challenge for this application is to learn a projection operator that takes an arbitrary 3D voxel model and produces a latent vector on the shape manifold from which a similar and realistic shape can be generated. We develop algorithms for this and other steps of the SNAP processing pipeline and integrate them into a simple modeling tool. Experiments with these algorithms and tool suggest that GANs provide a promising approach to computer-assisted interactive modeling.

Published at International Conference on 3D Vision 2017 (http://irc.cs.sdu.edu.cn/3dv/index.html)

References in corpus (5)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Autoencoding beyond pixels using a learned similarity metric
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Multi-view Convolutional Neural Networks for 3D Shape Recognition
• Unsupervised Learning of 3D Structure from Images

Cited by in corpus (7)

• 3D Sketching using Multi-View Deep Volumetric Prediction
• Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
• Coherent Point Drift Networks: Unsupervised Learning of Non-Rigid Point Set Registration
• CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality
• Hierarchical Detail Enhancing Mesh-Based Shape Generation with 3D Generative Adversarial Network
• DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces
• End-to-end 3D shape inverse rendering of different classes of objects from a single input image