Publications (16)
VideoMatGen: PBR Materials through Joint Generative Modeling
Jon Hasselgren, Zheng Zeng, Milos Hasan +1
We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a t…
Extracting Neural Materials from Multi-view Images
Kim Youwang, Jon Hasselgren, Peter Kocsis +3
Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challengin…
Noise2Noise: Learning Image Restoration without Clean Data
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren +4
We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusio…
DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Ruofan Liang, Zan Gojcic, Huan Ling +8
Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport…
Appearance-Driven Automatic 3D Model Simplification
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen +2
We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, includ…
VideoMat: Extracting PBR Materials from Video Diffusion Models
Jacob Munkberg, Zian Wang, Ruofan Liang +2
We leverage finetuned video diffusion models, intrinsic decomposition of videos, and physically-based differentiable rendering to generate high quality materials for 3D models give…
Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
Tianchang Shen, Jacob Munkberg, Jon Hasselgren +7
This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly co…
Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising
Jon Hasselgren, Nikolai Hofmann, Jacob Munkberg
Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filt…
Extracting Triangular 3D Models, Materials, and Lighting From Images
Jacob Munkberg, Jon Hasselgren, Tianchang Shen +5
We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, wh…
Edify 3D: Scalable High-Quality 3D Asset Generation
NVIDIA, :, Maciej Bala +22
We introduce Edify 3D, an advanced solution designed for high-quality 3D asset generation. Our method first synthesizes RGB and surface normal images of the described object at mul…
Toward Richer Material Generation via Procedural Data Enhancement
Yunchen Yu, Jacob Munkberg, Jon Hasselgren +3
Generative models for material creation are fundamentally limited by the quality and expressivity of available training data. Simple physically based rendering (PBR) materials, whi…
AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidance
Xilong Zhou, Bao-Huy Nguyen, Zheng Zeng +6
Video generative models have achieved remarkable progress in synthesizing photorealistic video sequences. However, enabling broader and more creative downstream applications requir…
Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes
Zian Wang, Tianchang Shen, Jun Gao +6
Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion. Recent NeRF based methods…
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Kai He, Ruofan Liang, Jacob Munkberg +7
We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-t…
Generative Detail Enhancement for Physically Based Materials
Saeed Hadadan, Benedikt Bitterli, Tizian Zeltner +6
We present a tool for enhancing the detail of physically based materials using an off-the-shelf diffusion model and inverse rendering. Our goal is to enhance the visual fidelity of…
TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions
Or Perel, Hassan Abu Alhaija, Zian Wang +4
We introduce TRON, a rendering framework that combines 3D Gaussian ray tracing with neural rendering to enable realistic and controllable rendering of real-world 3D scenes under no…