38 citations · 106 across the 18 of their papers we have counts for
13 papers · 1 filter
MultEval: Supporting Collaborative Alignment for LLM-as-a-Judge Evaluation Criteria
Charles Chiang, Simret Gebreegziabher, Annalisa Szymanski +6
LLM-as-a-judge approaches have emerged as a scalable solution for evaluating model behaviors, yet they rely on evaluation criteria often created by a single individual, embedding t…
The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes
Simret Araya Gebreegziabher, Yukun Yang, Charles Chiang +7
Large Language Model (LLM)-powered web GUI agents are increasingly automating everyday online tasks. Despite their popularity, little is known about how users' preferences and valu…
Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges
Hyo Jin Do, Zahra Ashktorab, Jasmina Gajcin +5
The LLM-as-a-judge paradigm enables flexible, user-defined evaluation, but its effectiveness is often limited by the scarcity of diverse, representative data for refining criteria.…
Who's Sorry Now: User Preferences Among Rote, Empathic, and Explanatory Apologies from LLM Chatbots
Zahra Ashktorab, Alessandra Buccella, Jason D'Cruz +6
As chatbots driven by large language models (LLMs) are increasingly deployed in everyday contexts, their ability to recover from errors through effective apologies is critical to m…
EvalAssist: A Human-Centered Tool for LLM-as-a-Judge
Zahra Ashktorab, Werner Geyer, Michael Desmond +6
With the broad availability of large language models and their ability to generate vast outputs using varied prompts and configurations, determining the best output for a given tas…
Togedule: Scheduling Meetings with Large Language Models and Adaptive Representations of Group Availability
Jaeyoon Song, Zahra Ashktorab, Thomas W. Malone
Scheduling is a perennial-and often challenging-problem for many groups. Existing tools are mostly static, showing an identical set of choices to everyone, regardless of the curren…
Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
Zahra Ashktorab, Michael Desmond, Qian Pan +7
Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes ti…
Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations
Zahra Ashktorab, Qian Pan, Werner Geyer +5
We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content-grounded data generation, focusing on the use of Large Language Models…
Human-Centered Design Recommendations for LLM-as-a-Judge
Qian Pan, Zahra Ashktorab, Michael Desmond +5
Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-qua…
Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback
Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil +4
Millions of users come to online peer counseling platforms to seek support. However, studies show that online peer support groups are not always as effective as expected, largely d…
Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness
Zahra Ashktorab, Benjamin Hoover, Mayank Agarwal +4
Mitigating algorithmic bias is a critical task in the development and deployment of machine learning models. While several toolkits exist to aid machine learning practitioners in a…
Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interface
Michael Desmond, Zahra Ashktorab, Michelle Brachman +8
Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides w…
Group Chat Ecology in Enterprise Instant Messaging: How Employees Collaborate Through Multi-User Chat Channels on Slack
Dakuo Wang, Haoyu Wang, Mo Yu +2
Despite the long history of studying instant messaging usage, we know very little about how today's people participate in group chat channels and interact with others inside a real…