Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview
arXiv:2405.05526 · doi:10.1016/j.engappai.2024.109685
Abstract
Neural Radiance Fields (NeRF) have emerged as a powerful paradigm for 3D scene representation, offering high-fidelity renderings and reconstructions from a set of sparse and unstructured sensor data. In the context of autonomous robotics, where perception and understanding of the environment are pivotal, NeRF holds immense promise for improving performance. In this paper, we present a comprehensive survey and analysis of the state-of-the-art techniques for utilizing NeRF to enhance the capabilities of autonomous robots. We especially focus on the perception, localization and navigation, and decision-making modules of autonomous robots and delve into tasks crucial for autonomous operation, including 3D reconstruction, segmentation, pose estimation, simultaneous localization and mapping (SLAM), navigation and planning, and interaction. Our survey meticulously benchmarks existing NeRF-based methods, providing insights into their strengths and limitations. Moreover, we explore promising avenues for future research and development in this domain. Notably, we discuss the integration of advanced techniques such as 3D Gaussian splatting (3DGS), large language models (LLM), and generative AIs, envisioning enhanced reconstruction efficiency, scene understanding, decision-making capabilities. This survey serves as a roadmap for researchers seeking to leverage NeRFs to empower autonomous robots, paving the way for innovative solutions that can navigate and interact seamlessly in complex environments.
32 pages, 5 figures, 8 tables
References in corpus (60)
- Rethinking Atrous Convolution for Semantic Image Segmentation
- ORB-SLAM: a Versatile and Accurate Monocular SLAM System
- ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
- ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- ShapeNet: An Information-Rich 3D Model Repository
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- The Replica Dataset: A Digital Replica of Indoor Spaces
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
- NeRF--: Neural Radiance Fields Without Known Camera Parameters
- DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
- Object Goal Navigation using Goal-Oriented Semantic Exploration
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
- Shap-E: Generating Conditional 3D Implicit Functions
- NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)
- Decomposing NeRF for Editing via Feature Field Distillation
- PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
- Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction
- NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction with Implicit Neural Representations
- VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
- HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
- DNeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
- IR-MCL: Implicit Representation-Based Online Global Localization
- SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection
- Offline Visual Representation Learning for Embodied Navigation
- CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations
- Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields
- CATNIPS: Collision Avoidance Through Neural Implicit Probabilistic Scenes
- ActiveRMAP: Radiance Field for Active Mapping And Planning
- Point-SLAM: Dense Neural Point Cloud-based SLAM
- How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey
- iDF-SLAM: End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking
- NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance
- MIRA: Mental Imagery for Robotic Affordances
- BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields
- Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
- DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAM
- No RL, No Simulation: Learning to Navigate without Navigating
- Neural Implicit Dense Semantic SLAM
- Hi-Map: Hierarchical Factorized Radiance Field for High-Fidelity Monocular Dense Mapping
- GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction
- AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
- Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
- GenS: Generalizable Neural Surface Reconstruction from Multi-View Images
- Evaluating Neural Radiance Fields (NeRFs) for 3D Plant Geometry Reconstruction in Field Conditions
- Implicit Learning of Scene Geometry from Poses for Global Localization
- 3D Reconstruction and New View Synthesis of Indoor Environments based on a Dual Neural Radiance Field
- Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Review
- BID-NeRF: RGB-D image pose estimation with inverted Neural Radiance Fields
- SegNeRF: 3D Part Segmentation with Neural Radiance Fields
- DNS SLAM: Dense Neural Semantic-Informed SLAM
- Neural Radiance Field in Autonomous Driving: A Survey
- Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors
- NeuV-SLAM: Fast Neural Multiresolution Voxel Optimization for RGBD Dense SLAM
- Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions
- SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects
- ID-Pose: Sparse-view Camera Pose Estimation by Inverting Diffusion Models
- PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields