39 citations · 169 across the 14 of their papers we have counts for
6 papers · 1 filter
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
Yang Li, Gang Li, Xin Zhou +2
User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detec…
Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
Forrest Huang, Gang Li, Xin Zhou +2
The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however…
Large-Scale Modeling of Mobile User Click Behaviors Using Deep Learning
Xin Zhou, Yang Li
Modeling tap or click sequences of users on a mobile device can improve our understandings of interaction behavior and offers opportunities for UI optimization by recommending next…
HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
Mingyuan Zhong, Gang Li, Peggy Chi +1
We present HelpViz, a tool for generating contextual visual mobile tutorials from text-based instructions that are abundant on the web. HelpViz transforms text instructions to grap…
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Bryan Wang, Gang Li, Xin Zhou +3
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be usef…
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
Yang Li, Si Si, Gang Li +2
Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or…