ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Abstract

Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.

16 pages, 7 figures

References in corpus (9)

• nocaps: novel object captioning at scale
• End-to-end Projector Photometric Compensation
• Animated Stickies: Fast Video Projection Mapping onto a Markerless Plane through a Direct Closed-Loop Alignment
• End-to-end Full Projector Compensation
• DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
• Projection Mapping under Environmental Lighting by Replacing Room Lights with Heterogeneous Projectors
• LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization
• DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
• GS-ProCams: Gaussian Splatting-based Projector-Camera Systems