3 citations · 4 across the 2 of their papers we have counts for
7 papers
UIClip: A Data-driven Model for Assessing User Interface Design
Jason Wu, Yi-Hao Peng, Amanda Li +3
User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-…
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Keen You, Haotian Zhang, Eldon Schoop +5
Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact e…
Towards Automated Accessibility Report Generation for Mobile Apps
Amanda Swearngin, Jason Wu, Xiaoyi Zhang +8
Many apps have basic accessibility issues, like missing labels or low contrast. Automated tools can help app developers catch basic issues, but can be laborious or require writing…
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
Yue Jiang, Eldon Schoop, Amanda Swearngin +1
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of…
Never-ending Learning of User Interfaces
Jason Wu, Rebecca Krosnick, Eldon Schoop +3
Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most…
Screen Correspondence: Mapping Interchangeable Elements between UIs
Jason Wu, Amanda Swearngin, Xiaoyi Zhang +2
Understanding user interface (UI) functionality is a useful yet challenging task for both machines and people. In this paper, we investigate a machine learning approach for screen…