4 papers · 1 filter
Mapping the Design Space of User Experience for Computer Use Agents
Ruijia Cheng, Jenny T. Liang, Eldon Schoop +1
Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with thes…
Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes
Jenny T. Liang, Titus Barik, Jeffrey Nichols +2
Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underl…
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
Jazbo Beason, Ruijia Cheng, Eldon Schoop +1
It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of mult…
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
Zhuohao Jerry Zhang, Eldon Schoop, Jeffrey Nichols +2
With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has…