A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection

Abstract

A substantial portion of offensive content on social media is directed towards women. Since the approaches for general offensive content detection face a challenge in detecting misogynistic content, it requires solutions tailored to address offensive content against women. To this end, we propose a novel multimodal framework for the detection of misogynistic and sexist content. The framework comprises three modules: the Multimodal Attention module (MANM), the Graph-based Feature Reconstruction Module (GFRM), and the Content-specific Features Learning Module (CFLM). The MANM employs adaptive gating-based multimodal context-aware attention, enabling the model to focus on relevant visual and textual information and generating contextually relevant features. The GFRM module utilizes graphs to refine features within individual modalities, while the CFLM focuses on learning text and image-specific features such as toxicity features and caption features. Additionally, we curate a set of misogynous lexicons to compute the misogyny-specific lexicon score from the text. We apply test-time augmentation in feature space to better generalize the predictions on diverse inputs. The performance of the proposed approach has been evaluated on two multimodal datasets, MAMI and MMHS150K, with 11,000 and 13,494 samples, respectively. The proposed method demonstrates an average improvement of 10.17% and 8.88% in macro-F1 over existing methods on the MAMI and MMHS150K datasets, respectively.

Published in Information Processing & Management

References in corpus (7)

• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Robust Real-World Image Super-Resolution against Adversarial Attacks
• Kosmos-2: Grounding Multimodal Large Language Models to the World
• Low Resource Text Classification with ULMFit and Backtranslation
• A Multimodal Framework for Depression Detection during Covid-19 via Harvesting Social Media: A Novel Dataset and Method
• KisanQRS: A Deep Learning-based Automated Query-Response System for Agricultural Decision-Making
• User-Aware Multilingual Abusive Content Detection in Social Media

Cited by in corpus (3)

• A Hybrid Similarity-Aware Graph Neural Network with Transformer for Node Classification
• Emotion-aware Dual Cross-Attentive Neural Network with Label Fusion for Stance Detection in Misinformative Social Media Content
• A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension