2 citations · 7 across the 41 of their papers we have counts for
14 papers · 1 filter
Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding
Koki Natsumi, Hiroyuki Deguchi, Yusuke Sakai +2
Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candida…
Reliability Crisis of Reference-free Metrics for Grammatical Error Correction
Takumi Goto, Yusuke Sakai, Taro Watanabe
Reference-free evaluation metrics for grammatical error correction (GEC) have achieved high correlation with human judgments. However, these metrics are not designed to evaluate ad…
Multilingual Dialogue Generation and Localization with Dialogue Act Scripting
Justin Vasselli, Eunike Andriani Kardinata, Yusuke Sakai +1
Non-English dialogue datasets are scarce, and models are often trained or evaluated on translations of English-language dialogues, an approach which can introduce artifacts that re…
SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala
Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda +6
Large Language Models (LLMs) demonstrate impressive general knowledge and reasoning abilities, yet their evaluation has predominantly focused on global or anglocentric subjects, of…
Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark
Chihiro Taguchi, Seng Mai, Keita Kurabe +4
Multilingual machine translation (MT) benchmarks play a central role in evaluating the capabilities of modern MT systems. Among them, the FLORES+ benchmark is widely used, offering…
Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability
Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on in…
IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator
Yusuke Sakai, Takumi Goto, Taro Watanabe
We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the qu…
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries
Haruki Sakajo, Yusuke Ide, Justin Vasselli +4
Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages. Existing approaches that utiliz…
gec-metrics: A Unified Library for Grammatical Error Correction Evaluation
Takumi Goto, Yusuke Sakai, Taro Watanabe
We introduce gec-metrics, a library for using and developing grammatical error correction (GEC) evaluation metrics through a unified interface. Our library enables fair system comp…
Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies
Kazuki Hayashi, Shintaro Ozaki, Yusuke Sakai +2
Large-scale Vision-Language Models (LVLMs) are being deployed in real-world settings that require visual inference. As capabilities improve, applications in navigation, education,…
TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation
Shintaro Ozaki, Tomoyuki Jinno, Kazuki Hayashi +6
When generating images from prompts that include specific entities, the model must retain as much entity-specific knowledge as possible. However, the number of entities is almost c…
Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?
Takumi Goto, Yusuke Sakai, Taro Watanabe
One of the goals of automatic evaluation metrics in grammatical error correction (GEC) is to rank GEC systems such that it matches human preferences. However, current automatic eva…
Tonguescape: Exploring Language Models Understanding of Vowel Articulation
Haruki Sakajo, Yusuke Sakai, Hidetaka Kamigaito +1
Vowels are primarily characterized by tongue position. Humans have discovered these features of vowel articulation through their own experience and explicit objective observation s…
Can Impressions of Music be Extracted from Thumbnail Images?
Takashi Harada, Takehiro Motomitsu, Katsuhiko Hayashi +2
In recent years, there has been a notable increase in research on machine learning models for music retrieval and generation systems that are capable of taking natural language sen…