10 citations · 33 across the 39 of their papers we have counts for
51 papers
Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
Users increasingly rely on large language models (LLMs) to analyse collections of independent inputs, such as product reviews or social media posts, and return a single aggregated…
Cubic Quasinormal Modes in the Black-Hole Ringdown
Neev Khera, Huan Yang
We formulate and numerically solve the nonlinear Teukolsky equations through third order, obtaining a first-principles calculation of cubic quasinormal-mode amplitudes for Kerr bla…
WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution
Yi Zhou, Kiamehr Rezaee, Danushka Bollegala +2
Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing…
CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text
Mohammad Reza Modarres, Armin Tourajmehr, Yadollah Yaghoobzadeh +1
Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered. We examine how reliably LLMs evaluate sho…
Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment o…
Exploring State Tracking Capabilities of Large Language Models
Kiamehr Rezaee, Jose Camacho-Collados, Mohammad Taher Pilehvar
Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on…
Pun Unintended: LLMs and the Illusion of Humor Understanding
Alessandro Zangari, Matteo Marcuzzo, Andrea Albarelli +2
Puns are a form of humorous wordplay that exploits polysemy and phonetic similarity. While LLMs have shown promise in detecting puns, we show in this paper that their understanding…
MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
Matteo Marcuzzo, Alessandro Zangari, Andrea Albarelli +2
As LLMs excel on standard reading comprehension benchmarks, attention is shifting toward evaluating their capacity for complex abstract reasoning and inference. Literature-based be…
Synthia: Scalable Grounded Persona Generation from Social Media Data
Vahid Rahimzadeh, Erfan Moosavi Monazzah, Mohammad Taher Pilehvar +1
Persona-driven simulations are increasingly used in computational social science, yet their validity critically depends on the fidelity of the underlying personas. Constructing vir…
Graph of Attacks: Improved Black-Box and Interpretable Jailbreaks for LLMs
Mohammad Akbar-Tajari, Mohammad Taher Pilehvar, Mohammad Mahmoody
The challenge of ensuring Large Language Models (LLMs) align with societal standards is of increasing interest, as these models are still prone to adversarial jailbreaks that bypas…
Gender Encoding Patterns in Pretrained Language Model Representations
Mahdi Zakizadeh, Mohammad Taher Pilehvar
Gender bias in pretrained language models (PLMs) poses significant social and ethical challenges. Despite growing awareness, there is a lack of comprehensive investigation into how…
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian
Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh +2
Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLM…
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets
Mahdi Zakizadeh, Mohammad Taher Pilehvar
Accurately measuring gender stereotypical bias in language models is a complex task with many hidden aspects. Current benchmarks have underestimated this multifaceted challenge and…
FarExStance: Explainable Stance Detection for Farsi
Majid Zarharan, Maryam Hashemi, Malika Behroozrazegh +3
We introduce FarExStance, a new dataset for explainable stance detection in Farsi. Each instance in this dataset contains a claim, the stance of an article or social media post tow…
NormXLogit: The Head-on-Top Never Lies
Sina Abbasi, Mohammad Reza Modarres, Mohammad Taher Pilehvar
With new large language models (LLMs) emerging frequently, it is important to consider the potential value of model-agnostic approaches that can provide interpretability across a v…
RepMatch: Quantifying Cross-Instance Similarities in Representation Space
Mohammad Reza Modarres, Sina Abbasi, Mohammad Taher Pilehvar
Advances in dataset analysis techniques have enabled more sophisticated approaches to analyzing and characterizing training data instances, often categorizing data based on attribu…
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Junho Myung, Nayeon Lee, Yi Zhou +19
Large language models (LLMs) often lack culture-specific knowledge of daily life, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LL…
Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models
Majid Zarharan, Pascal Wullschleger, Babak Behkam Kia +2
This paper presents a comprehensive analysis of explainable fact-checking through a series of experiments, focusing on the ability of large language models to verify public health…
DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias
Mahdi Zakizadeh, Kaveh Eskandari Miandoab, Mohammad Taher Pilehvar
Numerous debiasing techniques have been proposed to mitigate the gender bias that is prevalent in pretrained language models. These are often evaluated on datasets that check the e…
DecompX: Explaining Transformers Decisions by Propagating Token Decomposition
Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh +2
An emerging solution for explaining Transformer-based models is to use vector-based analysis on how the representations are formed. However, providing a faithful vector-based expla…
An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning
Mohammad Akbar-Tajari, Sara Rajaee, Mohammad Taher Pilehvar
Parameter-efficient fine-tuning approaches have recently garnered a lot of attention. Having considerably lower number of trainable weights, these methods can bring about scalabili…
Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similarities
Ali Modarressi, Hossein Amirkhani, Mohammad Taher Pilehvar
Several proposals have been put forward in recent years for improving out-of-distribution (OOD) performance through mitigating dataset biases. A popular workaround is to train a ro…
BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning
Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi +3
Current pre-trained language models rely on large datasets for achieving state-of-the-art performance. However, past research has shown that not all examples in a dataset are equal…
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference
Sara Rajaee, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar
It has been shown that NLI models are usually biased with respect to the word-overlap between premise and hypothesis; they take this feature as a primary cue for predicting the ent…