Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Abstract

We present Kaolin, a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable 3D modules for use in deep learning systems. With functionality to load and preprocess several popular 3D datasets, and native functions to manipulate meshes, pointclouds, signed distance functions, and voxel grids, Kaolin mitigates the need to write wasteful boilerplate code. Kaolin packages together several differentiable graphics modules including rendering, lighting, shading, and view warping. Kaolin also supports an array of loss functions and evaluation metrics for seamless evaluation and provides visualization functionality to render the 3D results. Importantly, we curate a comprehensive model zoo comprising many state-of-the-art 3D deep learning architectures, to serve as a starting point for future research endeavours. Kaolin is available as open-source software at https://github.com/NVIDIAGameWorks/kaolin/.

Kaolin is available as open-source software at https://github.com/NVIDIAGameWorks/kaolin/

References in corpus (7)

• Inductive Representation Learning on Large Graphs
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• MeshCNN: A Network with an Edge
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects

Cited by in corpus (23)

• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Differentiable Rendering: A Survey
• Accelerating 3D Deep Learning with PyTorch3D
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction
• gradSim: Differentiable simulation for system identification and visuomotor control
• Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework using Visual and Depth Cues
• 3D Shape Reconstruction from Vision and Touch
• Neural-IMLS: Self-supervised Implicit Moving Least-Squares Network for Surface Reconstruction
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
• RobustPointSet: A Dataset for Benchmarking Robustness of Point Cloud Classifiers
• MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
• Towards Differentiable Rendering for Sidescan Sonar Imagery
• Self-Sampling for Neural Point Cloud Consolidation
• Interactive Annotation of 3D Object Geometry using 2D Scribbles
• Pose2RGBD. Generating Depth and RGB images from absolute positions
• Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
• Extending DeepSDF for automatic 3D shape retrieval and similarity transform estimation
• 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations
• Physical world assistive signals for deep neural network classifiers -- neither defense nor attack
• Torch-Points3D: A Modular Multi-Task Frameworkfor Reproducible Deep Learning on 3D Point Clouds