3D Gaussian Splatting with Deferred Reflection

Abstract

The advent of neural and Gaussian-based radiance field methods have achieved great success in the field of novel view synthesis. However, specular reflection remains non-trivial, as the high frequency radiance field is notoriously difficult to fit stably and accurately. We present a deferred shading method to effectively render specular reflection with Gaussian splatting. The key challenge comes from the environment map reflection model, which requires accurate surface normal while simultaneously bottlenecks normal estimation with discontinuous gradients. We leverage the per-pixel reflection gradients generated by deferred shading to bridge the optimization process of neighboring Gaussians, allowing nearly correct normal estimations to gradually propagate and eventually spread over all reflective objects. Our method significantly outperforms state-of-the-art techniques and concurrent work in synthesizing high-quality specular reflection effects, demonstrating a consistent improvement of peak signal-to-noise ratio (PSNR) for both synthetic and real-world scenes, while running at a frame rate almost identical to vanilla Gaussian splatting.

References in corpus (3)

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
• Neural Point Catacaustics for Novel-View Synthesis of Reflections

Cited by in corpus (8)

• When Gaussian Meets Surfel: Ultra-fast High-fidelity Radiance Field Rendering
• GS-ROR${}^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction
• TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians
• 2DGH: 2D Gaussian-Hermite Splatting for High-quality Rendering and Better Geometry Features
• MSGS: Multispectral 3D Gaussian Splatting
• From Volume Rendering to 3D Gaussian Splatting: Theory and Applications
• Intrinsic decomposition and editing of 3D Gaussian splats
• Gaussian Splatting with Reflections Guided by What is Seen