10 papers
Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models
Fali Wang, Ali Al-Lawati, Iliyas Bektas +5
Graph reasoning provides a promising testbed for evaluating the reasoning ability of large language models (LLMs), as graph instances can be programmatically generated, structurall…
DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects
Jason Lucas, Matt Murtagh, Ali Al-Lawati +3
Harmful content detectors, particularly disinformation classifiers, are predominantly developed and evaluated on Standard American English (SAE), leaving their robustness to dialec…
LLM Benchmark Datasets Should Be Contamination-Resistant
Ali Al-Lawati, Jason Lucas, Dongwon Lee +1
Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretr…
Moltbook Moderation: Uncovering Hidden Intent Through Multi-Turn Dialogue
Ali Al-Lawati, Nafis Tripto, Abolfazl Ansari +3
The emergence of multi-agent systems introduces novel moderation challenges that extend beyond content filtering. Agents with malicious intent may contribute harmful content that a…
Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attacks
Ali Al-Lawati, Suhang Wang
The growing adoption of multimodal Retrieval-Augmented Generation (mRAG) pipelines for vision-centric tasks (e.g., visual QA) introduces important privacy challenges. In particular…
BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages
Jason Lucas, Matt Murtagh-White, Adaku Uchendu +6
Multilingual falsehoods threaten information integrity worldwide, yet detection benchmarks remain confined to English or a few high-resource languages, leaving low-resource linguis…