10 papers
Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It
Jonghyun Song, Haewon Park, Jeonghoon Shim +2
As LLM agents decide on users' behalf which product to buy, which hotel to book, or which paper to cite, a preference for items from certain sources (the sites or services they com…
AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks
Woojung Song, Hoyeol Yang, Jeonghoon Shim +4
Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tas…
Agents' Overreliance on Unreliable Tools
Hoyeol Yang, Woojung Song, Taewon Kim +3
LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the righ…
ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Woojung Song, Nalim Kim, Sangjun Song +3
Role-playing language agents (RPLAs) simulate specific characters and personas across applications such as entertainment, companionship, interactive storytelling, and education. Fa…
Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models
Jonggeun Lee, Woojung Song, Jongwook Han +2
Small language models (SLMs) enable scalable tool-augmented multi-agent systems where multiple SLMs handle subtasks orchestrated by a powerful coordinator. However, they struggle w…
Quantifying Data Contamination in Psychometric Evaluations of LLMs
Jongwook Han, Woojung Song, Jonggeun Lee +1
Recent studies apply psychometric questionnaires to Large Language Models (LLMs) to assess high-level psychological constructs such as values, personality, moral foundations, and d…
Non-Collaborative User Simulators for Tool Agents
Jeonghoon Shim, Woojung Song, Cheyon Jin +2
Tool agents interact with users through multi-turn dialogues to accomplish various tasks. Recent studies have adopted user simulation methods to develop these agents in multi-turn…
Human Psychometric Questionnaires Mischaracterize LLM Behavior
Woojung Song, Dongmin Choi, Yoonah Park +3
We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open…
Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
Sungjib Lim, Woojung Song, Eun-Ju Lee +1
As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A c…
Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items
Jongwook Han, Dongmin Choi, Woojung Song +2
The importance of benchmarks for assessing the values of language models has been pronounced due to the growing need of more authentic, human-aligned responses. However, existing b…