15 citations · 42 across the 36 of their papers we have counts for
36 papers · 1 filter
How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?
Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +2
While recent studies have explored human behavior and preference simulation using large language models (LLMs), it remains unclear how well LLMs can simulate subjective evaluations…
Large Language Models as Modal Models in Linguistics
Haruto Suzuki, Saku Sugawara
The rapid advancement of large language models (LLMs) has intensified debates about their significance for linguistic theory. These debates are commonly divided into three position…
A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models
Rei Emura, Saku Sugawara
Language models (LMs) behave more like humans when their cognitive resources are restricted, particularly in predicting sentence processing costs such as reading times. However, it…
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
Akira Kawabata, Saku Sugawara
Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing method…
Trust Stack for Mental Health AI: A Survey of Calibration across Human, Interaction, and AI Layers
Xin Sun, Yue Su, Yifan Mo +9
Language-based AI is increasingly deployed for mental health support, yet trust is evaluated in interdisciplinary but operationally misaligned ways: NLP and AI work measures robust…
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models
Miyu Oba, Saku Sugawara
Recent work has examined language models from a linguistic perspective to better understand how they acquire language. Most existing benchmarks focus on judging grammatical accepta…
Investigating Learner-Aware Design of LLM-Generated Educational Feedback
Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3
Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answ…
TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?
Yiwei Liu, Emma Jane Pretty, Jiahao Huang +1
While recent studies explore Large Language Models' (LLMs) performance on Theory of Mind (ToM) reasoning tasks, research on ToM abilities that require more nuanced social context i…
Specification-Aware Machine Translation and Evaluation for Purpose Alignment
Yoko Kayano, Saku Sugawara
In professional settings, translation is guided by communicative goals and client needs, often formalized as specifications. While existing evaluation frameworks acknowledge the im…
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +1
Automatic evaluation of generative tasks using large language models faces challenges due to ambiguous criteria. Although automatic checklist generation is a potentially promising…
Rationale-Aware Answer Verification by Pairwise Self-Evaluation
Akira Kawabata, Saku Sugawara
Answer verification identifies correct solutions among candidates generated by large language models (LLMs). Current approaches typically train verifier models by labeling solution…
Can Language Models Induce Grammatical Knowledge from Indirect Evidence?
Miyu Oba, Yohei Oseki, Akiyo Fukatsu +4
What kinds of and how much data is necessary for language models to induce grammatical knowledge to judge sentence acceptability? Recent language models still have much room for im…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
MoreHopQA: More Than Multi-hop Reasoning
Julian Schnitzler, Xanh Ho, Jiahao Huang +3
Most existing multi-hop datasets are extractive answer datasets, where the answers to the questions can be extracted directly from the provided context. This often leads models to…
What Makes Language Models Good-enough?
Daiki Asami, Saku Sugawara
Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural fea…
PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments
Daiki Asami, Saku Sugawara
What makes a presupposition of an utterance -- information taken for granted by its speaker -- different from other pragmatic inferences such as an entailment is projectivity (e.g.…
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Akira Kawabata, Saku Sugawara
To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning…
Probing Physical Reasoning with Counter-Commonsense Context
Kazushi Kondo, Saku Sugawara, Akiko Aizawa
In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; t…
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Saku Sugawara, Shun Tsugita
Natural language understanding (NLU) studies often exaggerate or underestimate the capabilities of systems, thereby limiting the reproducibility of their findings. These erroneous…
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara +1
To explain the predicted answers and evaluate the reasoning abilities of models, several studies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA)…
Which Shortcut Solution Do Question Answering Models Prefer to Learn?
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. QA models that have learned shortcut so…
Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
Question answering (QA) models are shown to be insensitive to large perturbations to inputs; that is, they make correct and confident predictions even when given largely perturbed…
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
Johannes Mario Meissner, Saku Sugawara, Akiko Aizawa
Debiasing language models from unwanted behaviors in Natural Language Understanding tasks is a topic with rapidly increasing interest in the NLP community. Spurious statistical cor…
Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
Extractive question answering (QA) models tend to exploit spurious correlations to make predictions when a training set has unintended biases. This tendency results in models not b…