Accelerating 3D Deep Learning with PyTorch3D
arXiv:2007.08501
Abstract
Deep learning has significantly improved 2D image recognition. Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition. However despite growing interest, 3D deep learning remains relatively underexplored. We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable. We address these challenges by introducing PyTorch3D, a library of modular, efficient, and differentiable operators for 3D deep learning. It includes a fast, modular differentiable renderer for meshes and point clouds, enabling analysis-by-synthesis approaches. Compared with other differentiable renderers, PyTorch3D is more modular and efficient, allowing users to more easily extend it while also gracefully scaling to large meshes and images. We compare the PyTorch3D operators and renderer with other implementations and demonstrate significant speed and memory improvements. We also use PyTorch3D to improve the state-of-the-art for unsupervised 3D mesh and point cloud prediction from 2D images on ShapeNet. PyTorch3D is open-source and we hope it will help accelerate research in 3D deep learning.
tech report
References in corpus (7)
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Fast Graph Representation Learning with PyTorch Geometric
- Places: An Image Database for Deep Scene Understanding
- Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
- Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
- GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects