HAMMER: Heterogeneous, Multi-Robot Semantic Gaussian Splatting
arXiv:2501.14147 · doi:10.1109/LRA.2025.3575235
Abstract
3D Gaussian Splatting offers expressive scene reconstruction, modeling a broad range of visual, geometric, and semantic information. However, efficient real-time map reconstruction with data streamed from multiple robots and devices remains a challenge. To that end, we propose HAMMER, a server-based collaborative Gaussian Splatting method that leverages widely available ROS communication infrastructure to generate 3D, metric-semantic maps from asynchronous robot data-streams with no prior knowledge of initial robot positions and varying on-device pose estimators. HAMMER consists of (i) a frame alignment module that transforms local SLAM poses and image data into a global frame and requires no prior relative pose knowledge, and (ii) an online module for training semantic 3DGS maps from streaming data. HAMMER handles mixed perception modes, adjusts automatically for variations in image pre-processing among different devices, and distills CLIP semantic codes into the 3D scene for open-vocabulary language queries. In our real-world experiments, HAMMER creates higher-fidelity maps (2x) compared to competing baselines and is useful for downstream tasks, such as semantic goal-conditioned navigation (e.g., "go to the couch"). Accompanying content available at hammer-project.github.io.
References in corpus (11)
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- Robot Operating System 2: Design, Architecture, and Uses In The Wild
- KISS-ICP: In Defense of Point-to-Point ICP -- Simple, Accurate, and Robust Registration If Done the Right Way
- The Replica Dataset: A Digital Replica of Indoor Spaces
- Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
- DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning
- Project Aria: A New Tool for Egocentric Multi-Modal AI Research
- GauU-Scene: A Scene Reconstruction Benchmark on Large Scale 3D Reconstruction Dataset Using Gaussian Splatting
- Distributed NeRF Learning for Collaborative Multi-Robot Perception
- MAGiC-SLAM: Multi-Agent Gaussian Globally Consistent SLAM
- SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps