Zero-shot detection of buildings in mobile LiDAR using Language Vision Model

Abstract

Recent advances have demonstrated that Language Vision Models (LVMs) surpass the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision tasks, motivating attempts to apply LVMs to three-dimensional (3D) data. While LVMs are efficient and effective in addressing various downstream 2D vision tasks without training, they face significant challenges when it comes to point clouds, a representative format for representing 3D data. It is more difficult to extract features from 3D data and there are challenges due to large data sizes and the cost of the collection and labelling, resulting in a notably limited availability of datasets. Moreover, constructing LVMs for point clouds is even more challenging due to the requirements for large amounts of data and training time. To address these issues, our research aims to 1) apply the Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2) experiment with synthetic data to evaluate its effectiveness in bridging the gap between synthetic and real-world data domains. Our approach exhibited high performance with an accuracy of 0.96, an IoU of 0.85, precision of 0.92, recall of 0.91, and an F1 score of 0.92, confirming its potential. However, challenges such as occlusion problems and pixel-level overlaps of multi-label points during spherical image generation remain to be addressed in future studies.

7 pages, 6 figures, conference

References in corpus (12)

• Flamingo: a Visual Language Model for Few-Shot Learning
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Reproducible scaling laws for contrastive language-image learning
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
• A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
• Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
• SynthCity: A large scale synthetic point cloud
• OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding
• Learning Object-Language Alignments for Open-Vocabulary Object Detection
• OpenMask3D: Open-Vocabulary 3D Instance Segmentation
• PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
• Transfer Learning from Synthetic to Real LiDAR Point Cloud for Semantic Segmentation

Cited by in corpus (1)

• V-RoAst: Visual Road Assessment. Can VLM be a Road Safety Assessor Using the iRAP Standard?