Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
arXiv:2101.11103 · doi:10.1145/3411764.3445049
Abstract
Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec's key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.
Accepted to CHI Conference on Human Factors in Computing Systems (CHI 2021)
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning
- VASTA: A Vision and Language-assisted Smartphone Task Automation System
- Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Cited by in corpus (20)
- Structure-aware Visualization Retrieval
- Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
- Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development
- MineXR: Mining Personalized Extended Reality Interfaces
- Avgust: Automating Usage-Based Test Generation from Videos of App Executions
- Boosting GUI Prototyping with Diffusion Models
- Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
- AQuA: Automated Question-Answering in Software Tutorial Videos with Visual Anchors
- GUing: A Mobile GUI Search Engine using a Vision-Language Model
- Automatic Macro Mining from Interaction Traces at Scale
- Leveraging Multimodal LLM for Inspirational User Interface Search
- Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
- DiLogics: Creating Web Automation Programs With Diverse Logics
- Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
- AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding
- Understanding Mobile GUI: from Pixel-Words to Screen-Sentences
- A Need-finding Study for Understanding Text Entry in Smartphone App Usage
- Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
- Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
- CanvasVAE: Learning to Generate Vector Graphic Documents