Image-guided topic modeling for interpretable privacy classification

Abstract

Predicting and explaining the private information contained in an image in human-understandable terms is a complex and contextual task. This task is challenging even for large language models. To facilitate the understanding of privacy decisions, we propose to predict image privacy based on a set of natural language content descriptors. These content descriptors are associated with privacy scores that reflect how people perceive image content. We generate descriptors with our novel Image-guided Topic Modeling (ITM) approach. ITM leverages, via multimodality alignment, both vision information and image textual descriptions from a vision language model. We use the ITM-generated descriptors to learn a privacy predictor, Priv$\times$ITM, whose decisions are interpretable by design. Our Priv$\times$ITM classifier outperforms the reference interpretable method by 5 percentage points in accuracy and performs comparably to the current non-interpretable state-of-the-art model.

Paper accepted at the eXCV Workshop at ECCV 2024. Supplementary material included. Code available at https://github.com/idiap/itm

References in corpus (7)

• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Zero-Bias Deep Learning for Accurate Identification of Internet of Things (IoT) Devices
• Dynamic Deep Multi-modal Fusion for Image Privacy Prediction
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
• PEAK: Explainable Privacy Assistant through Automated Knowledge Extraction
• Human-interpretable and deep features for image privacy classification