3 citations · 3 across the 6 of their papers we have counts for
5 papers · 1 filter
Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability
Krishnapriya Vishnubhotla, Hillary Dawkins, Isar Nejadgholi +1
Adapting foundation large language models to a user's task or preferred style through fine-tuning can result in compromising the model's safety. Previous works examined the effects…
LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij +1
We evaluate the consistency of automated judges in conducting a multi-dimensional safety evaluation in a reference-free setup. Our results indicate that Large Language Models are u…
Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory
Isar Nejadgholi, Masoud Kianpour, Krishnapriya Vishnubhotla +1
Tremendous efforts have been put into evaluating the inclusivity and effectiveness of AI systems across cultures. However, the cultural capabilities considered in much of the liter…
The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories
Xi Yu Huang, Krishnapriya Vishnubhotla, Frank Rudzicz
The improved generative capabilities of large language models have made them a powerful tool for creative writing and storytelling. It is therefore important to quantitatively unde…
SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages
Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla +14
We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phen…