39 citations · 178 across the 15 of their papers we have counts for
15 papers
UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset
Peitong Duan, Chin-yi Chen, Gang Li +2
Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation,…
Automatic Macro Mining from Interaction Traces at Scale
Forrest Huang, Gang Li, Tao Li +1
Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile inte…
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Xi Chen, Josip Djolonga, Piotr Padlewski +40
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training t…
PLay: Parametrically Conditioned Layout Generation using Latent Diffusion
Chin-Yi Cheng, Forrest Huang, Gang Li +1
Layout design is an important task in various design fields, including user interface, document, and graphic design. As this task requires tedious manual effort by designers, prior…
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Gang Li, Yang Li
Mobile UI understanding is important for enabling various interaction tasks such as UI automation and accessibility. Previous mobile UI modeling often depends on the view hierarchy…
Enabling Conversational Interaction with Mobile UI using Large Language Models
Bryan Wang, Gang Li, Yang Li
Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typical…
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Eldon Schoop, Xin Zhou, Gang Li +3
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view h…
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale
Gang Li, Gilles Baechler, Manuel Tragut +1
The layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, h…
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
Yang Li, Gang Li, Xin Zhou +2
User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detec…
Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
Forrest Huang, Gang Li, Xin Zhou +2
The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however…
HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
Mingyuan Zhong, Gang Li, Peggy Chi +1
We present HelpViz, a tool for generating contextual visual mobile tutorials from text-based instructions that are abundant on the web. HelpViz transforms text instructions to grap…
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Bryan Wang, Gang Li, Xin Zhou +3
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be usef…
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
Yang Li, Si Si, Gang Li +2
Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or…
Spacewalker: Rapid UI Design Exploration Using Lightweight Markup Enhancement and Crowd Genetic Programming
Mingyuan Zhong, Gang Li, Yang Li
User interface design is a complex task that involves designers examining a wide range of options. We present Spacewalker, a tool that allows designers to rapidly search a large de…
Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements
Yang Li, Gang Li, Luheng He +3
Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descripti…