12 citations · 13 across the 13 of their papers we have counts for
5 papers · 1 filter
Write Once, Check Many: Reusable Ledgers for Post-Hoc Policy Compliance Auditing of Agent Trajectories
Ashwin Nagarajan, Gursimran Panesar, Alicia Yang +3
Auditing LLM agents against policies is critical to ensure safety but requires rereading long trajectories once per policy. Can we instead record each trajectory once and reuse tha…
RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
Aashiq Muhamed, Leonardo F. R. Ribeiro, Markus Dreyer +2
The ability of language models in RAG systems to selectively refuse to answer based on flawed context is critical for safety, yet remains a significant failure point. Our large-sca…
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language
Mena Attia, Aashiq Muhamed, Mai Alkhamissi +2
We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figur…
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation
Aashiq Muhamed
RAG systems enhance LLMs by incorporating external knowledge, which is crucial for domains that demand factual accuracy and up-to-date information. However, evaluating the multifac…
An In-depth Look at Gemini's Language Abilities
Syeda Nahida Akter, Zichun Yu, Aashiq Muhamed +6
The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper,…