5 papers
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina +1
Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script langu…
Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities
Rajvee Sheth, Samridhi Raj Sinha, Mahavir Patil +2
Amidst the rapid advances of large language models (LLMs), most LLMs still struggle with mixed-language inputs, limited Codeswitching (CSW) datasets, and evaluation biases, which h…
Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models
Samridhi Raj Sinha, Rajvee Sheth, Abhishek Upperwal +1
The rapid evolution of Large Language Models' has underscored the need for evaluation frameworks that are globally applicable, flexible, and modular, and that support a wide range…
COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing
Rajvee Sheth, Himanshu Beniwal, Mayank Singh
We introduce COMI-LINGUA, the largest manually annotated Hindi-English code-mixed dataset, comprising 125K+ high-quality instances across five core NLP tasks: Matrix Language Ident…
COMMENTATOR: A Code-mixed Multilingual Text Annotation Framework
Rajvee Sheth, Shubh Nisar, Heenaben Prajapati +2
As the NLP community increasingly addresses challenges associated with multilingualism, robust annotation tools are essential to handle multilingual datasets efficiently. In this p…