MoralCLIP: Contrastive Alignment of Vision-and-Language Representations with Moral Foundations Theory

Abstract

Recent advances in vision-language models have enabled rich semantic understanding across modalities. However, these encoding methods lack the ability to interpret or reason about the moral dimensions of content-a crucial aspect of human cognition. In this paper, we address this gap by introducing MoralCLIP, a novel embedding representation method that extends multimodal learning with explicit moral grounding based on Moral Foundations Theory (MFT). Our approach integrates visual and textual moral cues into a unified embedding space, enabling cross-modal moral alignment. MoralCLIP is grounded on the multi-label dataset Social-Moral Image Database to identify co-occurring moral foundations in visual content. For MoralCLIP training, we design a moral data augmentation strategy to scale our annotated dataset to 15,000 image-text pairs labeled with MFT-aligned dimensions. Our results demonstrate that explicit moral supervision improves both unimodal and multimodal understanding of moral content, establishing a foundation for morally-aware AI systems capable of recognizing and aligning with human moral values.

Updated version: corresponds to the ACM MM '25 published paper and includes full appendix material

References in corpus (6)

• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Development and Validation of the Japanese Moral Foundations Dictionary
• Image Retrieval from Contextual Descriptions
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussions
• MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models