Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles with Label Propagation for Cross-Domain Adaptation

Abstract

Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. Here, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from multiple sensor modalities augments 3D point clouds and image segmentation masks while also generating an allocentric semantic map. Label propagation on the semantic map allows for sensor-specific adaptation with cross-modality and cross-domain supervision. Our system provides augmented semantic images and point clouds with $\approx$ 9 Hz. We evaluate the integrated system in real-world experiments in an urban environment and at a disaster test site.

arXiv comment35 pages, 18 figures, Accepted for Robotics and Autonomous Systems, Elsevier. arXiv admin note: substantial text overlap with arXiv:2108.06608

References in corpus (6)

• TensorFlow: A system for large-scale machine learning
• Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning
• SuMa++: Efficient LiDAR-based Semantic SLAM
• Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery
• Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping
• Semi-Supervised Semantic Mapping through Label Propagation with Semantic Texture Meshes