Cross-Modal Conceptualization in Bottleneck Models

Abstract

Concept Bottleneck Models (CBMs) assume that training examples (e.g., x-ray images) are annotated with high-level concepts (e.g., types of abnormalities), and perform classification by first predicting the concepts, followed by predicting the label relying on these concepts. The main difficulty in using CBMs comes from having to choose concepts that are predictive of the label and then having to label training examples with these concepts. In our approach, we adopt a more moderate assumption and instead use text descriptions (e.g., radiology reports), accompanying the images in training, to guide the induction of concepts. Our cross-modal approach treats concepts as discrete latent variables and promotes concepts that (1) are predictive of the label, and (2) can be predicted reliably from both the image and text. Through experiments conducted on datasets ranging from synthetic datasets (e.g., synthetic images with generated descriptions) to realistic medical imaging datasets, we demonstrate that cross-modal learning encourages the induction of interpretable concepts while also facilitating disentanglement. Our results also suggest that this guidance leads to increased robustness by suppressing the reliance on shortcut features.

arXiv commentAccepted at EMNLP 2023; camera-ready version

References in corpus (14)

• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• Categorical Reparameterization with Gumbel-Softmax
• Axiomatic Attribution for Deep Networks
• Understanding Black-box Predictions via Influence Functions
• e-SNLI: Natural Language Inference with Natural Language Explanations
• Object-Centric Learning with Slot Attention
• Concept Bottleneck Models
• Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias
• Post-hoc Concept Bottleneck Models
• Promises and Pitfalls of Black-Box Concept Learning Models
• Label-Free Concept Bottleneck Models
• A Closer Look at the Intervention Procedure of Concept Bottleneck Models
• Learning to Intervene on Concept Bottlenecks

Cited by in corpus (2)

• CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Models
• Leveraging Expert Input for Robust and Explainable AI-Assisted Lung Cancer Detection in Chest X-rays