121 citations · 231 across the 32 of their papers we have counts for
34 papers
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
LLM Judges Can Be Too Generous When There Is No Reference Answer
Chalamalasetti Kranti, Sowmya Vajjala
LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably…
LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij +1
We evaluate the consistency of automated judges in conducting a multi-dimensional safety evaluation in a reference-free setup. Our results indicate that Large Language Models are u…
ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation
Joseph Marvin Imperial, Junhong Liang, Belal Shoer +9
When a text is translated, does the translation retain the complexity of the original? We introduce ComplexityMT, a new challenge for assessing how text complexity and machine tran…
Translating Classical Poetry into Modern Prose
Chalamalasetti Kranti, Sowmya Vajjala
We built a dataset for the task of poem-to-prose translation from 13th-17th Century Telugu classical poetry to contemporary Telugu and English prose, which we call Padyam2Gadyam. T…
Improving Methodologies for LLM Evaluations Across Global Languages
Akriti Vij, Benjamin Chua, Darshini Ramiah +43
As frontier AI models are deployed globally, it is essential that their behaviour remains safe and reliable across diverse linguistic and cultural contexts. To examine how current…
Improving Methodologies for Agentic Evaluations Across Domains: Leakage of Sensitive Information, Fraud and Cybersecurity Threats
Ee Wei Seah, Yongsen Zheng, Naga Nikshith +67
The rapid rise of autonomous AI systems and advancements in agent capabilities are introducing new risks due to reduced oversight of real-world interactions. Yet agent testing rema…
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…
IndicGEC: Powerful Models, or a Measurement Mirage?
Sowmya Vajjala
In this paper, we report the results of the TeamNRC's participation in the BHASHA-Task 1 Grammatical Error Correction shared task https://github.com/BHASHA-Workshop/IndicGEC2025/ f…
MetricalARGS: A Taxonomy for Studying Metrical Poetry with LLMs
Chalamalasetti Kranti, Sowmya Vajjala
Prior NLP work studying poetry has focused primarily on automatic poem generation and summarization. Many languages have well-studied traditions of poetic meter which enforce const…
MATA: Mindful Assessment of the Telugu Abilities of Large Language Models
Chalamalasetti Kranti, Sowmya Vajjala
In this paper, we introduce MATA, a novel evaluation dataset to assess the ability of Large Language Models (LLMs) in Telugu language, comprising 729 carefully curated multiple-cho…
Test Set Quality in Multilingual LLM Evaluation
Chalamalasetti Kranti, Gabriel Bernier-Colborne, Yvan Gauthier +1
Several multilingual benchmark datasets have been developed in a semi-automatic manner in the recent past to measure progress and understand the state-of-the-art in the multilingua…
The Problem with Safety Classification is not just the Models
Sowmya Vajjala
Studying the robustness of Large Language Models (LLMs) to unsafe behaviors is an important topic of research today. Building safety classification models or guard models, which ar…
Opportunities and Challenges of LLMs in Education: An NLP Perspective
Sowmya Vajjala, Bashar Alhafni, Stefano Bannò +2
Interest in the role of large language models (LLMs) in education is increasing, considering the new opportunities they offer for teaching, learning, and assessment. In this paper,…
UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment
Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden +16
We introduce UniversalCEFR, a large-scale multilingual and multidimensional dataset of texts annotated with CEFR (Common European Framework of Reference) levels in 13 languages. To…
Does Synthetic Data Help Named Entity Recognition for Low-Resource Languages?
Gaurav Kamath, Sowmya Vajjala
Named Entity Recognition(NER) for low-resource languages aims to produce robust systems for languages where there is limited labeled training data available, and has been an area o…
Text Classification in the LLM Era -- Where do we stand?
Sowmya Vajjala, Shwetali Shimangaud
Large Language Models revolutionized NLP and showed dramatic performance improvements across several tasks. In this paper, we investigated the role of such language models in text…
LLMs in Education: Novel Perspectives, Challenges, and Opportunities
Bashar Alhafni, Sowmya Vajjala, Stefano Bannò +2
The role of large language models (LLMs) in education is an increasing area of interest today, considering the new opportunities they offer for teaching, learning, and assessment.…
Annotation Errors and NER: A Study with OntoNotes 5.0
Gabriel Bernier-Colborne, Sowmya Vajjala
Named Entity Recognition (NER) is a well-studied problem in NLP. However, there is much less focus on studying NER datasets, compared to developing new NER models. In this paper, w…
Dravidian language family through Universal Dependencies lens
Taraka Rama, Sowmya Vajjala
The Universal Dependencies (UD) project aims to create a cross-linguistically consistent dependency annotation for multiple languages, to facilitate multilingual NLP. It currently…
Scope Ambiguities in Large Language Models
Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala +1
Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insight…
A Multilingual Evaluation of NER Robustness to Adversarial Inputs
Akshay Srinivasan, Sowmya Vajjala
Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its…
Automatic Text Simplification of News Articles in the Context of Public Broadcasting
Diego Maupomé, Fanny Rancourt, Thomas Soulas +8
This report summarizes the work carried out by the authors during the Twelfth Montreal Industrial Problem Solving Workshop, held at Université de Montréal in August 2022. The team…
What do we Really Know about State of the Art NER?
Sowmya Vajjala, Ramya Balasubramaniam
Named Entity Recognition (NER) is a well researched NLP task and is widely used in real world NLP scenarios. NER research typically focuses on the creation of new ways of training…