3 citations · 6 across the 7 of their papers we have counts for
11 papers
The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
Samee Arif, Furkan Danisman, Angana Borah +6
Large Language Models (LLMs) increasingly operate across culturally diverse settings, yet existing evaluations largely measure cultural behavior at the output level. We introduce C…
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs
Naihao Deng, Samee Arif, Shuaichen Chang +2
Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benchmarks such as BBQ have become…
The Age of Curiosity Meets the Age of AI: Benchmarking Child Safety in Large Language Models
Samee Arif, Angana Borah, Rada Mihalcea
Children increasingly have access to Large Language Models (LLMs), which may expose them to responses that are developmentally inappropriate or require age-sensitive safety, guidan…
One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety
Samee Arif, Naihao Deng, Zhijing Jin +1
Large Language Models (LLMs) are trained to refuse harmful requests, yet they remain vulnerable to jailbreak attacks that exploit weaknesses in conversational safety mechanisms. We…
From Press to Pixels: Evolving Urdu Text Recognition
Samee Arif, Sualeha Farid
This paper presents a comparative analysis of Large Language Models (LLMs) and traditional Optical Character Recognition (OCR) systems on Urdu newspapers, addressing challenges pos…
With a Grain of SALT: Are LLMs Fair Across Social Dimensions?
Samee Arif, Zohaib Khan, Maaidah Kaleem +3
This paper presents a systematic analysis of biases in open-source Large Language Models (LLMs), across gender, religion, and race. Our study evaluates bias in smaller-scale Llama…
Kahaani: A Multimodal Co-Creative Storytelling System
Samee Arif, Muhammad Saad Haroon, Aamina Jamal Khan +3
This paper introduces Kahaani, a multimodal, co-creative storytelling system that leverages Generative Artificial Intelligence, designed for children to address the challenge of su…
WER We Stand: Benchmarking Urdu ASR Models
Samee Arif, Sualeha Farid, Aamina Jamal Khan +3
This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless…
The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation
Samee Arif, Sualeha Farid, Abdul Hameed Azeemi +2
This paper presents a novel methodology for generating synthetic Preference Optimization (PO) datasets using multi-model workflows. We evaluate the effectiveness and potential of t…
Generalists vs. Specialists: Evaluating Large Language Models for Urdu
Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza +1
In this paper, we compare general-purpose models, GPT-4-Turbo and Llama-3-8b, with special-purpose models--XLM-Roberta-large, mT5-large, and Llama-3-8b--that have been fine-tuned o…
UQA: Corpus for Urdu Question Answering
Samee Arif, Sualeha Farid, Awais Athar +1
This paper introduces UQA, a novel dataset for question answering and text comprehension in Urdu, a low-resource language with over 70 million native speakers. UQA is generated by…