50 citations · 93 across the 14 of their papers we have counts for
14 papers
SentinelBench: A Benchmark for Long-Running Monitoring Agents
Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin +5
AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshin…
Improving User Interface Generation Models from Designer Feedback
Jason Wu, Amanda Swearngin, Arun Krishna Vajjala +3
Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation…
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
Zhuohao Jerry Zhang, Eldon Schoop, Jeffrey Nichols +2
With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has…
Misty: UI Prototyping Through Interactive Conceptual Blending
Yuwen Lu, Alan Leung, Amanda Swearngin +2
UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these exampl…
Automated Code Fix Suggestions for Accessibility Issues in Mobile Apps
Forough Mehralian, Titus Barik, Jeff Nichols +1
Accessibility is crucial for inclusive app usability, yet developers often struggle to identify and fix app accessibility issues due to a lack of awareness, expertise, and inadequa…
UIClip: A Data-driven Model for Assessing User Interface Design
Jason Wu, Yi-Hao Peng, Amanda Li +3
User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-…
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Keen You, Haotian Zhang, Eldon Schoop +5
Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact e…
Towards Automated Accessibility Report Generation for Mobile Apps
Amanda Swearngin, Jason Wu, Xiaoyi Zhang +8
Many apps have basic accessibility issues, like missing labels or low contrast. Automated tools can help app developers catch basic issues, but can be laborious or require writing…
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
Yue Jiang, Eldon Schoop, Amanda Swearngin +1
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of…
AXNav: Replaying Accessibility Tests from Natural Language
Maryam Taeb, Amanda Swearngin, Eldon Schoop +3
Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious,…
Never-ending Learning of User Interfaces
Jason Wu, Rebecca Krosnick, Eldon Schoop +3
Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most…
Screen Correspondence: Mapping Interchangeable Elements between UIs
Jason Wu, Amanda Swearngin, Xiaoyi Zhang +2
Understanding user interface (UI) functionality is a useful yet challenging task for both machines and people. In this paper, we investigate a machine learning approach for screen…
Extracting Replayable Interactions from Videos of Mobile App Usage
Jieshan Chen, Amanda Swearngin, Jason Wu +3
Screen recordings of mobile apps are a popular and readily available way for users to share how they interact with apps, such as in online tutorial videos, user reviews, or as atta…
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin +9
Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortuna…