39 citations · 178 across the 15 of their papers we have counts for
9 papers · 1 filter
UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset
Peitong Duan, Chin-yi Chen, Gang Li +2
Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation,…
Automatic Macro Mining from Interaction Traces at Scale
Forrest Huang, Gang Li, Tao Li +1
Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile inte…
Enabling Conversational Interaction with Mobile UI using Large Language Models
Bryan Wang, Gang Li, Yang Li
Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typical…
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Eldon Schoop, Xin Zhou, Gang Li +3
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view h…
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale
Gang Li, Gilles Baechler, Manuel Tragut +1
The layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, h…
Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
Forrest Huang, Gang Li, Xin Zhou +2
The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however…
HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
Mingyuan Zhong, Gang Li, Peggy Chi +1
We present HelpViz, a tool for generating contextual visual mobile tutorials from text-based instructions that are abundant on the web. HelpViz transforms text instructions to grap…
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Bryan Wang, Gang Li, Xin Zhou +3
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be usef…
Spacewalker: Rapid UI Design Exploration Using Lightweight Markup Enhancement and Crowd Genetic Programming
Mingyuan Zhong, Gang Li, Yang Li
User interface design is a complex task that involves designers examining a wide range of options. We present Spacewalker, a tool that allows designers to rapidly search a large de…