Multimodal Detection of Bots on X (Twitter) using Transformers

Abstract

Although not all bots are malicious, the vast majority of them are responsible for spreading misinformation and manipulating the public opinion about several issues, i.e., elections and many more. Therefore, the early detection of bots is crucial. Although there have been proposed methods for detecting bots in social media, there are still substantial limitations. For instance, existing research initiatives still extract a large number of features and train traditional machine learning algorithms or use GloVe embeddings and train LSTMs. However, feature extraction is a tedious procedure demanding domain expertise. Also, language models based on transformers have been proved to be better than LSTMs. Other approaches create large graphs and train graph neural networks requiring in this way many hours for training and access to computational resources. To tackle these limitations, this is the first study employing only the user description field and images of three channels denoting the type and content of tweets posted by the users. Firstly, we create digital DNA sequences, transform them to 3d images, and apply pretrained models of the vision domain, including EfficientNet, AlexNet, VGG16, etc. Next, we propose a multimodal approach, where we use TwHIN-BERT for getting the textual representation of the user description field and employ VGG16 for acquiring the visual representation for the image modality. We propose three different fusion methods, namely concatenation, gated multimodal unit, and crossmodal attention, for fusing the different modalities and compare their performances. Finally, we present a qualitative analysis of the behavior of our best performing model. Extensive experiments conducted on the Cresci'17 and TwiBot-20 datasets demonstrate valuable advantages of our introduced approaches over state-of-the-art ones.

IEEE Transactions on Information Forensics and Security (Accepted)

References in corpus (14)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• The spread of low-credibility content by social bots
• BotOrNot: A System to Evaluate Social Bots
• Deep Neural Networks for Bot Detection
• The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race
• Scalable and Generalizable Social Bot Detection through Data Selection
• TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
• DNA-inspired online behavioral modeling and its application to spambot detection
• Social Fingerprinting: detection of spambot groups through DNA-inspired behavioral modeling
• BotRGCN: Twitter Bot Detection with Relational Graph Convolutional Networks
• Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media
• Social bot detection in the age of ChatGPT: Challenges and opportunities
• RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Search
• Detecting Dementia from Speech and Transcripts using Transformers

Cited by in corpus (1)

• A Graph Neural Architecture Search Approach for Identifying Bots in Social Media