AggCLIP: Aggregating CLIP for Zero-Shot Multimodal Anomaly Detection
arXiv:2611.24643 · doi:10.1145/3767308.3835551
Abstract
Zero-Shot Multimodal Anomaly Detection (ZSMAD) aims to identify both appearance and geometric defects in novel categories without relying on target-specific training data, making it especially valuable in real-world scenarios constrained by data scarcity, privacy concerns, or high annotation costs. While recent approaches successfully adapt vision-language models like CLIP to achieve this open-vocabulary generalization, they still face significant challenges: (i) severe geometric information loss during 3D-to-2D point cloud rendering; (ii) spatially noisy 2D feature maps produced by raw CLIP patch embeddings, compounded by shallow late-stage modality fusion; and (iii) severe cross-modal semantic misalignment due to decoupled processing streams. To address these limitations, we propose AggCLIP, a novel framework that adapts frozen CLIP representations for ZSMAD by explicitly incorporating local geometric structure and visual spatial context. Specifically, to accurately localize fine-grained defects, we design a Geometry-Aware Spatial Aggregation (GSA) module that refines back-projected 3D point features via density-adaptive neighborhood weighting, alongside a Context-Aware Spatial Aggregation (CSA) module that smooths multi-scale 2D patch embeddings using a lightweight trainable adapter. Furthermore, to prevent semantic drift between modalities, we introduce a Dual-Stream Prompt Contrastive (DSPC) loss, which aligns the abstract concepts of normality and abnormality across the point cloud and RGB streams. Extensive experiments demonstrate that the proposed AggCLIP framework achieves state-of-the-art performance on the MVTec 3D-AD and Eyecandies benchmarks under both one-vs-rest and cross-dataset zero-shot settings, notably surpassing the strongest competitive methods by up to 2.9% in pixel-level AUPRO.
arXiv comment13 pages, 4 figures, 6 tables. Author version of the ACM Multimedia 2026 paper
References in corpus (8)
- Learning Transferable Visual Models From Natural Language Supervision
- Open3D: A Modern Library for 3D Data Processing
- The MVTec 3D-AD Dataset for Unsupervised 3D Anomaly Detection and Localization
- AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection
- AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection
- MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images
- Incomplete Multimodal Industrial Anomaly Detection via Cross-Modal Distillation
- Dinomaly2: A Unified Framework for Unsupervised Image Anomaly Detection