15 citations · 42 across the 36 of their papers we have counts for
39 papers
How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?
Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +2
While recent studies have explored human behavior and preference simulation using large language models (LLMs), it remains unclear how well LLMs can simulate subjective evaluations…
Large Language Models as Modal Models in Linguistics
Haruto Suzuki, Saku Sugawara
The rapid advancement of large language models (LLMs) has intensified debates about their significance for linguistic theory. These debates are commonly divided into three position…
A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models
Rei Emura, Saku Sugawara
Language models (LMs) behave more like humans when their cognitive resources are restricted, particularly in predicting sentence processing costs such as reading times. However, it…
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
Akira Kawabata, Saku Sugawara
Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing method…
Trust Stack for Mental Health AI: A Survey of Calibration across Human, Interaction, and AI Layers
Xin Sun, Yue Su, Yifan Mo +9
Language-based AI is increasingly deployed for mental health support, yet trust is evaluated in interdisciplinary but operationally misaligned ways: NLP and AI work measures robust…
Eyes Can't Always Tell: Fusing Eye Tracking and User Priors for User Modeling under AI Advice Conditions
Xin Sun, Shu Wei, Ting Pan +5
Modeling users' cognitive states (e.g., cognitive load and decision confidence) is essential for building adaptive AI in high-stakes decision-making. While eye tracking provides no…
Seeing the Reasoning: How LLM Rationales Influence User Trust and Decision-Making in Factual Verification Tasks
Xin Sun, Shu Wei, Jos A Bosch +3
Large Language Models (LLMs) increasingly show reasoning rationales alongside their answers, turning "reasoning" into a user-interface element. While step-by-step rationales are ty…
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models
Miyu Oba, Saku Sugawara
Recent work has examined language models from a linguistic perspective to better understand how they acquire language. Most existing benchmarks focus on judging grammatical accepta…
Investigating Learner-Aware Design of LLM-Generated Educational Feedback
Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3
Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answ…
TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?
Yiwei Liu, Emma Jane Pretty, Jiahao Huang +1
While recent studies explore Large Language Models' (LLMs) performance on Theory of Mind (ToM) reasoning tasks, research on ToM abilities that require more nuanced social context i…
Specification-Aware Machine Translation and Evaluation for Purpose Alignment
Yoko Kayano, Saku Sugawara
In professional settings, translation is guided by communicative goals and client needs, often formalized as specifications. While existing evaluation frameworks acknowledge the im…
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +1
Automatic evaluation of generative tasks using large language models faces challenges due to ambiguous criteria. Although automatic checklist generation is a potentially promising…
Rationale-Aware Answer Verification by Pairwise Self-Evaluation
Akira Kawabata, Saku Sugawara
Answer verification identifies correct solutions among candidates generated by large language models (LLMs). Current approaches typically train verifier models by labeling solution…
Can Language Models Induce Grammatical Knowledge from Indirect Evidence?
Miyu Oba, Yohei Oseki, Akiyo Fukatsu +4
What kinds of and how much data is necessary for language models to induce grammatical knowledge to judge sentence acceptability? Recent language models still have much room for im…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
MoreHopQA: More Than Multi-hop Reasoning
Julian Schnitzler, Xanh Ho, Jiahao Huang +3
Most existing multi-hop datasets are extractive answer datasets, where the answers to the questions can be extracted directly from the provided context. This often leads models to…
What Makes Language Models Good-enough?
Daiki Asami, Saku Sugawara
Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural fea…
PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments
Daiki Asami, Saku Sugawara
What makes a presupposition of an utterance -- information taken for granted by its speaker -- different from other pragmatic inferences such as an entailment is projectivity (e.g.…
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Akira Kawabata, Saku Sugawara
To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning…
Probing Physical Reasoning with Counter-Commonsense Context
Kazushi Kondo, Saku Sugawara, Akiko Aizawa
In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; t…
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Saku Sugawara, Shun Tsugita
Natural language understanding (NLU) studies often exaggerate or underestimate the capabilities of systems, thereby limiting the reproducibility of their findings. These erroneous…
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara +1
To explain the predicted answers and evaluate the reasoning abilities of models, several studies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA)…
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
Hongkuan Zhang, Saku Sugawara, Akiko Aizawa +3
Image captioning models require the high-level generalization ability to describe the contents of various images in words. Most existing approaches treat the image-caption pairs eq…
Which Shortcut Solution Do Question Answering Models Prefer to Learn?
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. QA models that have learned shortcut so…