2 citations · 7 across the 21 of their papers we have counts for
15 papers · 1 filter
Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Lester James V. Miranda, Ivan Vulić, Anna Korhonen
Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection…
Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
Ej Zhou, Caiqi Zhang, Tiancheng Hu +4
Confidence calibration, the alignment of a model's predicted confidence with its actual accuracy, is crucial for the reliable deployment of Large Language Models (LLMs). However, t…
Quantifying Language Disparities in Multilingual Large Language Models
Songbo Hu, Ivan Vulić, Anna Korhonen
Results reported in large-scale multilingual evaluations are often fragmented and confounded by factors such as target languages, differences in experimental setups, and model choi…
Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes
Sharan Maiya, Yinhong Liu, Ramit Debnath +1
Large Language Models (LLMs) are often used as automated judges to evaluate text, but their effectiveness can be hindered by various unintentional biases. We propose using linear c…
TopViewRS: Vision-Language Models as Top-View Spatial Reasoners
Chengzu Li, Caiqi Zhang, Han Zhou +3
Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of `no…
Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
Han Zhou, Xingchen Wan, Yinhong Liu +3
Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language generation quality. In particular, pairwise LLM e…
Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?
Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen
Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. ICL has ga…
Self-Augmented In-Context Learning for Unsupervised Word Translation
Yaoyiran Li, Anna Korhonen, Ivan Vulić
Recent work has shown that, while large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups, they still…
DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models
Songbo Hu, Xiaobin Wang, Zhangdie Yuan +2
We present DIALIGHT, a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (ToD) systems which facilitates systematic evaluations and comparisons between ToD…
Large Language Models are Miscalibrated In-Context Learners
Chengzu Li, Han Zhou, Goran Glavaš +2
When adapting ICL with or without fine-tuning, we are curious about whether the instruction-tuned language model is able to achieve well-calibrated results without suffering from t…
SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU
Evgeniia Razumovskaia, Goran Glavaš, Anna Korhonen +1
Task-oriented dialogue (ToD) systems help users execute well-defined tasks across a variety of domains (e.g., or ), with their Nat…
Quantifying the Dialect Gap and its Correlates Across Languages
Anjali Kantharuban, Ivan Vulić, Anna Korhonen
Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), bu…
A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems
Songbo Hu, Han Zhou, Moy Yuan +5
Achieving robust language technologies that can perform well across the world's many languages is a central goal of multilingual NLP. In this work, we take stock of and empirically…
Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning
Han Zhou, Xingchen Wan, Ivan Vulić +1
Prompt-based learning has been an effective paradigm for large pretrained language models (LLM), enabling few-shot or even zero-shot learning. Black-box prompt search has received…
On Bilingual Lexicon Induction with Large Language Models
Yaoyiran Li, Anna Korhonen, Ivan Vulić
Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global…