A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
arXiv:2403.07469 · doi:10.1109/TCSVT.2023.3296889
Abstract
Three-Dimensional (3D) dense captioning is an emerging vision-language bridging task that aims to generate multiple detailed and accurate descriptions for 3D scenes. It presents significant potential and challenges due to its closer representation of the real world compared to 2D visual captioning, as well as complexities in data collection and processing of 3D point cloud sources. Despite the popularity and success of existing methods, there is a lack of comprehensive surveys summarizing the advancements in this field, which hinders its progress. In this paper, we provide a comprehensive review of 3D dense captioning, covering task definition, architecture classification, dataset analysis, evaluation metrics, and in-depth prosperity discussions. Based on a synthesis of previous literature, we refine a standard pipeline that serves as a common paradigm for existing methods. We also introduce a clear taxonomy of existing models, summarize technologies involved in different modules, and conduct detailed experiment analysis. Instead of a chronological order introduction, we categorize the methods into different classes to facilitate exploration and analysis of the differences and connections among existing techniques. We also provide a reading guideline to assist readers with different backgrounds and purposes in reading efficiently. Furthermore, we propose a series of promising future directions for 3D dense captioning by identifying challenges and aligning them with the development of related tasks, offering valuable insights and inspiring future research in this field. Our aim is to provide a comprehensive understanding of 3D dense captioning, foster further investigations, and contribute to the development of novel applications in multimedia and related domains.
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- ShapeNet: An Information-Rich 3D Model Repository
- nocaps: novel object captioning at scale
- Deep Learning Approaches on Image Captioning: A Review
- UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
- Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
- Show, Recall, and Tell: Image Captioning with Recall Mechanism
- D3Net: Densely connected multidilated DenseNet for music source separation
- Graph Neural Networks in Vision-Language Image Understanding: A Survey
- End-to-end Dense Video Captioning as Sequence Generation
- Contextual Modeling for 3D Dense Captioning on Point Clouds
- Learning Point-Language Hierarchical Alignment for 3D Visual Grounding