Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

Abstract

Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across $\sim$22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.

arXiv commentUIST'21

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• A Deep Reinforced Model for Abstractive Summarization
• VQA: Visual Question Answering
• Visual Storytelling
• Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
• Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements
• Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels