8 papers
Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information
Kaivalya Rawal, Daria Onitiu, Brent Mittelstadt +2
Explainable Artificial Intelligence (XAI) seeks to explain how an Artificial Intelligence (AI) system arrived at a particular decision. We propose ''Rule of Thumb'' (RoT) explanati…
The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Will Hawkins, Kaivalya Rawal, Jonathan Rystrøm +8
Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability…
Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments
Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel +22
As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchma…
AI-Mediated Communication Can Steer Collective Opinion
Stratis Tsirtsis, Kai Rawal, Chris Russell +2
Generative artificial intelligence (AI) is increasingly integrated into the online platforms where humans exchange opinions; large language models (LLMs) now polish users' posts on…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set
Kaivalya Rawal, Eoin Delaney, Zihao Fu +2
Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, expla…