1 citations · 1 across the 10 of their papers we have counts for
10 papers
Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluation
Luke Zhang, Justin Vasselli, Aditya Khan +2
We propose Dynamic Meta-Metrics (DMM), a framework for machine translation evaluation that learns source-sentence conditioned combinations of existing metrics. Rather than relying…
Multilingual Dialogue Generation and Localization with Dialogue Act Scripting
Justin Vasselli, Eunike Andriani Kardinata, Yusuke Sakai +1
Non-English dialogue datasets are scarce, and models are often trained or evaluated on translations of English-language dialogues, an approach which can introduce artifacts that re…
Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors
Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova +3
This shared task has aimed to assess pedagogical abilities of AI tutors powered by large language models (LLMs), focusing on evaluating the quality of tutor responses aimed at stud…
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries
Haruki Sakajo, Yusuke Ide, Justin Vasselli +4
Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages. Existing approaches that utiliz…
Measuring the Robustness of Reference-Free Dialogue Evaluation Systems
Justin Vasselli, Adam Nohejl, Taro Watanabe
Advancements in dialogue systems powered by large language models (LLMs) have outpaced the development of reliable evaluation metrics, particularly for diverse and creative respons…
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction
Takumi Goto, Justin Vasselli, Taro Watanabe
Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainabilit…
CoAM: Corpus of All-Type Multiword Expressions
Yusuke Ide, Joshua Tanner, Adam Nohejl +4
Multiword expressions (MWEs) refer to idiomatic sequences of multiple words. MWE identification, i.e., detecting MWEs in text, can play a key role in downstream tasks such as machi…
Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?
Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata +5
Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words even in the era of large language models (LLMs). Frequency in film subtitles…
How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments
Yusuke Ide, Yuto Nishida, Justin Vasselli +4
The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacce…
knn-seq: Efficient, Extensible kNN-MT Framework
Hiroyuki Deguchi, Hayate Hirano, Tomoki Hoshino +3
k-nearest-neighbor machine translation (kNN-MT) boosts the translation quality of a pre-trained neural machine translation (NMT) model by utilizing translation examples during deco…