39 citations · 169 across the 14 of their papers we have counts for
6 papers · 1 filter
Enabling Conversational Interaction with Mobile UI using Large Language Models
Bryan Wang, Gang Li, Yang Li
Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typical…
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Eldon Schoop, Xin Zhou, Gang Li +3
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view h…
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale
Gang Li, Gilles Baechler, Manuel Tragut +1
The layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, h…
Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
Forrest Huang, Gang Li, Xin Zhou +2
The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however…
HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
Mingyuan Zhong, Gang Li, Peggy Chi +1
We present HelpViz, a tool for generating contextual visual mobile tutorials from text-based instructions that are abundant on the web. HelpViz transforms text instructions to grap…
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Bryan Wang, Gang Li, Xin Zhou +3
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be usef…