Accelerating 3D Deep Learning with PyTorch3D

Abstract

Deep learning has significantly improved 2D image recognition. Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition. However despite growing interest, 3D deep learning remains relatively underexplored. We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable. We address these challenges by introducing PyTorch3D, a library of modular, efficient, and differentiable operators for 3D deep learning. It includes a fast, modular differentiable renderer for meshes and point clouds, enabling analysis-by-synthesis approaches. Compared with other differentiable renderers, PyTorch3D is more modular and efficient, allowing users to more easily extend it while also gracefully scaling to large meshes and images. We compare the PyTorch3D operators and renderer with other implementations and demonstrate significant speed and memory improvements. We also use PyTorch3D to improve the state-of-the-art for unsupervised 3D mesh and point cloud prediction from 2D images on ShapeNet. PyTorch3D is open-source and we hope it will help accelerate research in 3D deep learning.

tech report

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Fast Graph Representation Learning with PyTorch Geometric
• Differentiable Surface Splatting for Point-based Geometry Processing
• Places: An Image Database for Deep Scene Understanding
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects

Cited by in corpus (46)

• Score-Based Point Cloud Denoising
• NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
• gradSim: Differentiable simulation for system identification and visuomotor control
• Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
• Towards Universal Physical Attacks On Cascaded Camera-Lidar 3D Object Detection Models
• ADOP: Approximate Differentiable One-Pixel Point Rendering
• ClipMatrix: Text-controlled Creation of 3D Textured Meshes
• D3D-HOI: Dynamic 3D Human-Object Interactions from Videos
• DOVE: Learning Deformable 3D Objects by Watching Videos
• NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation
• Generative Models as Distributions of Functions
• Weakly-Supervised Multi-Face 3D Reconstruction
• Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
• To The Point: Correspondence-driven monocular 3D category reconstruction
• Differentiable Rendering with Perturbed Optimizers
• Revisiting Latent-Space Interpolation via a Quantitative Evaluation Framework
• E3D: Event-Based 3D Shape Reconstruction
• Shape As Points: A Differentiable Poisson Solver
• StyleUV: Diverse and High-fidelity UV Map Generative Model
• 3D Adversarial Attacks Beyond Point Cloud
• AutoFlow: Learning a Better Training Set for Optical Flow
• RobustPointSet: A Dataset for Benchmarking Robustness of Point Cloud Classifiers
• Recognizing Scenes from Novel Viewpoints
• Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding
• Roto-translated Local Coordinate Frames For Interacting Dynamical Systems
• Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Tracking
• LASR: Learning Articulated Shape Reconstruction from a Monocular Video
• Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering
• CodeNeRF: Disentangled Neural Radiance Fields for Object Categories
• DSC-PoseNet: Learning 6DoF Object Pose Estimation via Dual-scale Consistency
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates
• Tangent Space Backpropagation for 3D Transformation Groups
• Lifting 2D StyleGAN for 3D-Aware Face Generation
• EventHPE: Event-based 3D Human Pose and Shape Estimation
• A Deep Signed Directional Distance Function for Object Shape Representation
• Learning to Stylize Novel Views
• Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning
• Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
• Human Performance Capture from Monocular Video in the Wild
• Birds of a Feather: Capturing Avian Shape Models from Images
• Recurrently Estimating Reflective Symmetry Planes from Partial Pointclouds
• 3D Registration for Self-Occluded Objects in Context
• Physical world assistive signals for deep neural network classifiers -- neither defense nor attack
• Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface