Convolutional Generation of Textured 3D Meshes

Abstract

While recent generative models for 2D images achieve impressive visual results, they clearly lack the ability to perform 3D reasoning. This heavily restricts the degree of control over generated objects as well as the possible applications of such models. In this work, we bridge this gap by leveraging recent advances in differentiable rendering. We design a framework that can generate triangle meshes and associated high-resolution texture maps, using only 2D supervision from single-view natural images. A key contribution of our work is the encoding of the mesh and texture as 2D representations, which are semantically aligned and can be easily modeled by a 2D convolutional GAN. We demonstrate the efficacy of our method on Pascal3D+ Cars and CUB, both in an unconditional setting and in settings where the model is conditioned on class labels, attributes, and text. Finally, we propose an evaluation methodology that assesses the mesh and texture quality separately.

arXiv commentNeurIPS 2020, Oral presentation. Code at https://github.com/dariopavllo/convmesh

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Learning Representations and Generative Models for 3D Point Clouds
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• The Unusual Effectiveness of Averaging in GAN Training
• Controllable Text-to-Image Generation
• Visual Object Networks: Image Generation with Disentangled 3D Representation

Cited by in corpus (4)

• Discrete and continuous representations and processing in deep learning: Looking forward
• TM-NET: Deep Generative Networks for Textured Meshes
• Benchmarking the Robustness of Instance Segmentation Models
• Dual Contrastive Loss and Attention for GANs