565 citations · 1.3k across the 96 of their papers we have counts for
104 papers
DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries
Sahana Ramnath, Nima Chitsazan, Mingyang Zhou +8
Dialogues are a predominant mode of communication for humans, and it is immensely helpful to have automatically generated summaries of them (e.g., to revise key points discussed in…
Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling
Alaa Elsetohy, Sama Hadhoud, Haryo Akbarianto Wibowo +4
Multilingual benchmarks rarely test reasoning over culturally grounded premises: translated datasets keep English-centric scenarios, while culture-first datasets often lack control…
PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues
Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda +6
Code-switching is a widespread practice among the world's multilingual majority, yet few benchmarks accurately reflect its complexity in everyday communication. We present PingPong…
Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection
Tianyi Niu, Justin Chih-Yao Chen, Genta Indra Winata +6
Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unav…
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…
Vision Language Models are Confused Tourists
Patrick Amadeus Irawan, Ikhlasul Akmal Hanif, Muhammad Dehan Al Kautsar +3
Although the cultural dimension has been one of the key aspects in evaluating Vision-Language Models (VLMs), their ability to remain stable across diverse cultural inputs remains l…
Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs
Stefan Horoi, Sangwoo Cho, Supriyo Chakraborty +4
Task arithmetic is a powerful technique for transferring skills between Large Language Models (LLMs), but it often suffers from negative interference when models have diverged duri…
Optimizing Reasoning Efficiency through Prompt Difficulty Prediction
Bo Zhao, Berkcan Kapusuzoglu, Kartik Balasubramaniam +3
Reasoning language models perform well on complex tasks but are costly to deploy due to their size and long reasoning traces. We propose a routing approach that assigns each proble…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations
Patrick Amadeus Irawan, Ryandito Diandaru, Belati Jagad Bintang Syuhada +5
We introduce Entropy2Vec, a novel framework for deriving cross-lingual language representations by leveraging the entropy of monolingual language models. Unlike traditional typolog…
SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languages
Muhammad Dehan Al Kautsar, Aswin Candra, Muhammad Alif Al Hakim +6
Although numerous datasets have been developed to support dialogue systems, most existing chit-chat datasets overlook the cultural nuances inherent in natural human conversations.…
IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesian
Vanessa Rebecca Wiyono, David Anugraha, Ayu Purwarianti +1
Over 200 million people speak Indonesian, yet the language remains significantly underrepresented in preference-based research for large language models (LLMs). Most existing multi…
Language Surgery in Multilingual Large Language Models
Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong +6
Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigat…
Behind Maya: Building a Multilingual Vision Language Model
Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda +16
In recent times, we have seen a rapid development of large Vision-Language Models (VLMs). They have shown impressive results on academic benchmarks, primarily in widely spoken lang…
T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning
Amartya Chakraborty, Paresh Dashore, Nadia Bathaee +6
Large Language Models (LLMs) have demonstrated impressive capabilities as intelligent agents capable of solving complex problems. However, effective planning in scenarios involving…
R3: Robust Rubric-Agnostic Reward Models
David Anugraha, Zilu Tang, Lester James V. Miranda +5
Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models a…
What Causes Knowledge Loss in Multilingual Language Models?
Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti +1
Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that pr…
Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz +89
Southeast Asia (SEA) is a region of extraordinary linguistic and cultural diversity, yet it remains significantly underrepresented in vision-language (VL) research. This often resu…
RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferences
Hanyang Zhao, Haoxian Chen, Yucheng Guo +5
Traditional preference tuning methods for LLMs/Visual Generative Models often rely solely on reward model labeling, which can be opaque, offer limited insights into the rationale b…
TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoning
Frederikus Hudi, Genta Indra Winata, Ruochen Zhang +1
Reasoning is a fundamental capability of large language models (LLMs), enabling them to comprehend, analyze, and solve complex problems. In this paper, we introduce TextGames, an i…
Do Language Models Understand Honorific Systems in Javanese?
Mohammad Rifqi Farhansyah, Iwan Darmawan, Adryan Kusumawardhana +3
The Javanese language features a complex system of honorifics that vary according to the social status of the speaker, listener, and referent. Despite its cultural and linguistic s…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
Maya: An Instruction Finetuned Multilingual Multimodal Model
Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda +16
The rapid development of large Vision-Language Models (VLMs) has led to impressive results on academic benchmarks, primarily in widely spoken languages. However, significant gaps r…
A Multi-Agent Dual Dialogue System to Support Mental Health Care Providers
Onno P. Kampman, Ye Sheng Phang, Stanley Han +9
We introduce a general-purpose, human-in-the-loop dual dialogue system to support mental health care professionals. The system, co-designed with care providers, is conceptualized t…