VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points
arXiv:2410.17932 · doi:10.1145/3728302
Abstract
Recent advances in novel view synthesis have demonstrated impressive results in fast photorealistic scene rendering through differentiable point rendering, either via Gaussian Splatting (3DGS) [Kerbl and Kopanas et al. 2023] or neural point rendering [Aliev et al. 2020]. Unfortunately, these directions require either a large number of small Gaussians or expensive per-pixel post-processing for reconstructing fine details, which negatively impacts rendering performance. To meet the high performance demands of virtual reality (VR) systems, primitive or pixel counts therefore must be kept low, affecting visual quality. In this paper, we propose a novel hybrid approach based on foveated rendering as a promising solution that combines the strengths of both point rendering directions regarding performance sweet spots. Analyzing the compatibility with the human visual system, we find that using a low-detailed, few primitive smooth Gaussian representation for the periphery is cheap to compute and meets the perceptual demands of peripheral vision. For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output within the rendering budget. This combination also allows for synergistic method accelerations with point occlusion culling and reducing the demands on the neural network. Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration, and participants of a user study overwhelmingly preferred our method. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience. The project page can be found at: https://lfranke.github.io/vr_splatting
References in corpus (23)
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- 2D Gaussian Splatting for Geometrically Accurate Radiance Fields
- DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
- Neural Point Catacaustics for Novel-View Synthesis of Reflections
- VR-NeRF: High-Fidelity Virtualized Walkable Spaces
- Block-NeRF: Scalable Large Scene Neural View Synthesis
- Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs
- Learned Initializations for Optimizing Coordinate-Based Neural Representations
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction
- Foveated Rendering: Motivation, Taxonomy, and Research Directions
- View Synthesis with Sculpted Neural Points
- 3D Gaussian Splatting as Markov Chain Monte Carlo
- MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures
- VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering
- Immersive Neural Graphics Primitives
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
- SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes
- Software Rasterization of 2 Billion Points in Real Time
- Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View Synthesis
- PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation
- RPBG: Towards Robust Neural Point-based Graphics in the Wild
- D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video
- Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency