Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

Abstract

Neural graphics primitives, parameterized by fully connected neural networks, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA kernels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of $1920\times 1080$.

arXiv commentTo appear in ACM Transactions on Graphics (SIGGRAPH 2022). 15 pages, 13 figures, 3 tables

References in corpus (3)

• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
• Real-time Neural Radiance Caching for Path Tracing
• Neural Radiosity

Cited by in corpus (232)

• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields
• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
• AI and 6G into the Metaverse: Fundamentals, Challenges and Future Research Trends
• 3D Gaussian as a New Era: A Survey
• PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction
• HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video
• NeRF2: Neural Radio-Frequency Radiance Fields
• Reducing the Memory Footprint of 3D Gaussian Splatting
• NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction
• NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
• MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving
• Synthetic Datasets for Autonomous Driving: A Survey
• SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM
• High-fidelity 3D Reconstruction of Plants using Neural Radiance Field
• PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency
• Compact 3D Scene Representation via Self-Organizing Gaussian Grids
• NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction with Implicit Neural Representations
• Neural Point Catacaustics for Novel-View Synthesis of Reflections
• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
• AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels
• Real-World Robot Applications of Foundation Models: A Review
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting
• FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos
• 3D Gaussian Blendshapes for Head Avatar Animation
• Tracking and Mapping in Medical Computer Vision: A Review
• Efficient physics-informed neural networks using hash encoding
• Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomography
• Compressing Explicit Voxel Grid Representations: fast NeRFs become also small
• VR-NeRF: High-Fidelity Virtualized Walkable Spaces
• Fast Neural Representations for Direct Volume Rendering
• 3D Gaussian Splatting with Deferred Reflection
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
• H2-Mapping: Real-time Dense Mapping Using Hierarchical Hybrid Representation
• LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar
• LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives
• CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians
• Fast Learning Radiance Fields by Shooting Much Fewer Rays
• Fluid Simulation on Neural Flow Maps
• Real-Time Neural Appearance Models
• Spatiotemporal implicit neural representation for unsupervised dynamic MRI reconstruction
• 3DGS.zip: A survey on 3D Gaussian Splatting Compression Methods
• NPGA: Neural Parametric Gaussian Avatars
• PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
• Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery
• EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
• NeRF-NQA: No-Reference Quality Assessment for Scenes Generated by NeRF and Neural View Synthesis Methods
• Efficient Neural Implicit Representation for 3D Human Reconstruction
• SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions
• Random-Access Neural Compression of Material Textures
• SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection
• MetaSapiens: Real-Time Neural Rendering with Efficiency-Aware Pruning and Accelerated Foveated Rendering
• Reviewing Intelligent Cinematography: AI research for camera-based video production
• VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations
• Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases
• BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis
• RO-MAP: Real-Time Multi-Object Mapping with Neural Radiance Fields
• Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation
• Towards Open World NeRF-Based SLAM
• A Survey on Deep Generative 3D-aware Image Synthesis
• A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose
• CATNIPS: Collision Avoidance Through Neural Implicit Probabilistic Scenes
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• Don't Splat your Gaussians: Volumetric Ray-Traced Primitives for Modeling and Rendering Scattering and Emissive Media
• Variational Barycentric Coordinates
• Adaptively Placed Multi-Grid Scene Representation Networks for Large-Scale Data Visualization
• Unsupervised Continual Semantic Adaptation through Neural Rendering
• Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview
• MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal calibration
• A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis
• 3DGR-CT: Sparse-View CT Reconstruction with a 3D Gaussian Representation
• Online Neural Path Guiding with Normalized Anisotropic Spherical Gaussians
• Neural Parametric Mixtures for Path Guiding
• Operator Learning with Neural Fields: Tackling PDEs on General Geometries
• Rip-NeRF: Anti-aliasing Radiance Fields with Ripmap-Encoded Platonic Solids
• GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis
• ZS-SRT: An Efficient Zero-Shot Super-Resolution Training Method for Neural Radiance Fields
• UW-GS: Distractor-Aware 3D Gaussian Splatting for Enhanced Underwater Scene Reconstruction
• Crowd-Sourced NeRF: Collecting Data from Production Vehicles for 3D Street View Reconstruction
• Geometry-Aware Attenuation Learning for Sparse-View CBCT Reconstruction
• Implicit Conversion of Manifold B-Rep Solids by Neural Halfspace Representation
• Gaussian Splatting Decoder for 3D-aware Generative Adversarial Networks
• RGB-Only Gaussian Splatting SLAM for Unbounded Outdoor Scenes
• Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization
• HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction
• Cool-chic video: Learned video coding with 800 parameters
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
• N-BVH: Neural ray queries with bounding volume hierarchies
• Dynamic Structured Illumination Microscopy with a Neural Space-time Model
• Advances in Artificial Intelligence: A Review for the Creative Industries
• SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
• Continuous 3D Myocardial Motion Tracking via Echocardiography
• Deep Appearance Prefiltering
• TwinTex: Geometry-aware Texture Generation for Abstracted 3D Architectural Models
• Panoptic Vision-Language Feature Fields
• DynaMoN: Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields
• Geometric implicit neural representations for signed distance functions
• PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting
• SAT-NGP : Unleashing Neural Graphics Primitives for Fast Relightable Transient-Free 3D reconstruction from Satellite Imagery
• Neural Control Variates with Automatic Integration
• Explicit Correspondence Matching for Generalizable Neural Radiance Fields
• TetWeave: Isosurface Extraction using On-The-Fly Delaunay Tetrahedral Grids for Gradient-Based Mesh Optimization
• VIINTER: View Interpolation with Implicit Neural Representations of Images
• Bathymetric Surveying with Imaging Sonar Using Neural Volume Rendering
• HPC: Hierarchical Progressive Coding Framework for Volumetric Video
• SlimmeRF: Slimmable Radiance Fields
• MARF: The Medial Atom Ray Field Object Representation
• N-Dimensional Gaussians for Fitting of High Dimensional Functions
• Interactive Rendering of Relightable and Animatable Gaussian Avatars
• VRMM: A Volumetric Relightable Morphable Head Model
• Dynamic NeRFs for Soccer Scenes
• HI-SLAM2: Geometry-Aware Gaussian SLAM for Fast Monocular Scene Reconstruction
• Dynamic Appearance Particle Neural Radiance Field
• Laser: Efficient Language-Guided Segmentation in Neural Radiance Fields
• Multi-tiling Neural Radiance Field (NeRF) -- Geometric Assessment on Large-scale Aerial Datasets
• ARSGaussian: 3D Gaussian Splatting with LiDAR for Aerial Remote Sensing Novel View Synthesis
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians
• HAMMER: Heterogeneous, Multi-Robot Semantic Gaussian Splatting
• NeCA: 3D Coronary Artery Tree Reconstruction from Two 2D Projections via Neural Implicit Representation
• Factorized Motion Fields for Fast Sparse Input Dynamic View Synthesis
• Robust LiDAR-Camera Calibration with 2D Gaussian Splatting
• DenseSplat: Densifying Gaussian Splatting SLAM with Neural Radiance Prior
• Scalar Field Prediction on Meshes Using Interpolated Multi-Resolution Convolutional Neural Networks
• Neural Light Spheres for Implicit Image Stitching and View Synthesis
• Efficient Scene Appearance Aggregation for Level-of-Detail Rendering
• Localized Gaussian Splatting Editing with Contextual Awareness
• Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis
• EasyVolcap: Accelerating Neural Volumetric Video Research
• LiveNVS: Neural View Synthesis on Live RGB-D Streams
• FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees
• Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments
• Neural Field Convolutions by Repeated Differentiation
• When Gaussian Meets Surfel: Ultra-fast High-fidelity Radiance Field Rendering
• DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields
• Neural Distortion Fields for Spatial Calibration of Wide Field-of-View Near-Eye Displays
• GS-ROR${}^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction
• IntraSeismic: a coordinate-based learning approach to seismic inversion
• Efficient Graphics Representation with Differentiable Indirection
• PRoGS: Progressive Rendering of Gaussian Splats
• Acoustic Neural 3D Reconstruction Under Pose Drift
• MEIL-NeRF: Memory-Efficient Incremental Learning of Neural Radiance Fields
• As-Rigid-As-Possible Deformation of Gaussian Radiance Fields
• PG-NeuS: Robust and Efficient Point Guidance for Multi-View Neural Surface Reconstruction
• SILVR: A Synthetic Immersive Large-Volume Plenoptic Dataset
• Efficient Raycasting of Volumetric Depth Images for Remote Visualization of Large Volumes at High Frame Rates
• GaussEdit: Adaptive 3D Scene Editing with Text and Image Prompts
• DualMS: Implicit Dual-Channel Minimal Surface Optimization for Heat Exchanger Design
• SPARK: Self-supervised Personalized Real-time Monocular Face Capture
• Manifold Sampling for Differentiable Uncertainty in Radiance Fields
• Deep Neural Implicit Representation of Accessibility for Multi-Axis Manufacturing
• MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
• Radiance Surfaces: Optimizing Surface Representations with a 5D Radiance Field Loss
• IDDR-NGP: Incorporating Detectors for Distractor Removal with Instant Neural Radiance Field
• Neural Gaussian Scale-Space Fields
• Normal-guided Detail-Preserving Neural Implicit Function for High-Fidelity 3D Surface Reconstruction
• AquaNeRF: Neural Radiance Fields in Underwater Media with Distractor Removal
• Spline Deformation Field
• RNA: Relightable Neural Assets
• Virtualized 3D Gaussians: Flexible Cluster-based Level-of-Detail System for Real-Time Rendering of Composed Scenes
• Scalable and High-Quality Neural Implicit Representation for 3D Reconstruction
• Splat and Replace: 3D Reconstruction with Repetitive Elements
• MS-NeRF: Multi-Space Neural Radiance Fields
• Cutting-edge 3D reconstruction solutions for underwater coral reef images: A review and comparison
• P2I-NET: Mapping Camera Pose to Image via Adversarial Learning for New View Synthesis in Real Indoor Environments
• InfNeRF: Towards Infinite Scale NeRF Rendering with O(log n) Space Complexity
• Distributed Radiance Fields for Edge Video Compression and Metaverse Integration in Autonomous Driving
• StructGS: Adaptive Spherical Harmonics and Rendering Enhancements for Superior 3D Gaussian Splatting
• Low-Rank Adaptation of Neural Fields
• Active Scout: Multi-Target Tracking Using Neural Radiance Fields in Dense Urban Environments
• Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces
• FlatCAD: Fast Curvature Regularization of Neural SDFs for CAD Models
• MC-INR: Efficient Encoding of Multivariate Scientific Simulation Data using Meta-Learning and Clustered Implicit Neural Representations
• Methods and strategies for improving the novel view synthesis quality of neural radiation field
• Animating NeRFs from Texture Space: A Framework for Pose-Dependent Rendering of Human Performances
• 2DGH: 2D Gaussian-Hermite Splatting for High-quality Rendering and Better Geometry Features
• Improved 3D Gaussian Splatting of Unknown Spacecraft Structure Using Space Environment Illumination Knowledge
• Neural Parametric Surfaces for Shape Modeling
• GGAvatar: Reconstructing Garment-Separated 3D Gaussian Splatting Avatars from Monocular Video
• NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Prior
• Local Surface Parameterizations via Smoothed Geodesic Splines
• GS-ProCams: Gaussian Splatting-based Projector-Camera Systems
• Stylizing Sparse-View 3D Scenes with Hierarchical Neural Representation
• Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression
• EfficientHuman: Efficient Training and Reconstruction of Moving Human using Articulated 2D Gaussian
• Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splatting
• Surgical Neural Radiance Fields from One Image
• Anisotropic Neural Representation Learning for High-Quality Neural Rendering
• Directional Texture Editing for 3D Models
• Gabor Fields: Orientation-Selective Level-of-Detail for Volume Rendering
• Controllable 3D Object Generation with Single Image Prompt
• Animus3D: Text-driven 3D Animation via Motion Score Distillation
• Neural Rendering for Sensor Adaptation in 3D Object Detection
• Aerial Path Planning for Urban Geometry and Texture Co-Capture
• InFusionSurf: Refining Neural RGB-D Surface Reconstruction Using Per-Frame Intrinsic Refinement and TSDF Fusion Prior Learning
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting
• Text-based Animatable 3D Avatars with Morphable Model Alignment
• SOAP: Style-Omniscient Animatable Portraits
• MVImgNet2.0: A Larger-scale Dataset of Multi-view Images
• High-quality Animatable Eyelid Shapes from Lightweight Captures
• CharNeRF: 3D Character Generation from Concept Art
• Neural Stream Functions
• SeamlessNeRF: Stitching Part NeRFs with Gradient Propagation
• TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning
• GaussianFocus: Constrained Attention Focus for 3D Gaussian Splatting
• Simulation Based Reward Function Validation for Multi-Agent On Orbit Inspection
• HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks
• A Probability-guided Sampler for Neural Implicit Surface Rendering
• Vector Scaffolding: Inter-Scale Orchestration for Differentiable Image Vectorization
• ParkingTwin: Training-Free Streaming 3D Reconstruction for Parking-Lot Digital Twins
• MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion
• Learning High-Fidelity Robot Self-Model with Articulated 3D Gaussian Splatting
• Bayesian Monocular Depth Refinement via Neural Radiance Fields
• SharpNet: Enhancing MLPs to Represent Functions with Controlled Non-differentiability
• DiskChunGS: Large-Scale 3D Gaussian SLAM Through Chunk-Based Memory Management
• Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture
• As-Rigid-As-Possible Regularization for Implicit Surfaces
• Resolution Where It Counts: Hash-based GPU-Accelerated 3D Reconstruction via Variance-Adaptive Voxel Grids
• MicroSDF: Microfacet-Driven Hybrid Neural SDFs for Mixed-Reflectance Surface Reconstruction
• Splat-based Metal Artifact Reduction in Cone-Beam CT via Polychromatic Modeling
• Gaussian Splatting with Reflections Guided by What is Seen
• View-Adaptive Renderer for View-Consistent 2D-to-3D Generation
• Improving Machine Learning-Based Robot Self-Collision Checking with Input Positional Encoding
• Gaussian Building Mesh (GBM): Extract a Building's 3D Mesh with Google Earth and Gaussian Splatting
• Visual Tomography: Physically Faithful Volumetric Models of Partially Translucent Objects
• DiffTex: Differentiable Texturing for Architectural Proxy Models
• FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Images
• TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes
• 3D Gaussian Adaptive Reconstruction for Fourier Light-Field Microscopy
• GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images