VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
arXiv:2312.11190 · doi:10.1145/3654777.3676386
Abstract
Mobile task automation is an emerging field that leverages AI to streamline and optimize the execution of routine tasks on mobile devices, thereby enhancing efficiency and productivity. Traditional methods, such as Programming By Demonstration (PBD), are limited due to their dependence on predefined tasks and susceptibility to app updates. Recent advancements have utilized the view hierarchy to collect UI information and employed Large Language Models (LLM) to enhance task automation. However, view hierarchies have accessibility issues and face potential problems like missing object descriptions or misaligned structures. This paper introduces VisionTasker, a two-stage framework combining vision-based UI understanding and LLM task planning, for mobile task automation in a step-by-step manner. VisionTasker firstly converts a UI screenshot into natural language interpretations using a vision-based UI understanding approach, eliminating the need for view hierarchies. Secondly, it adopts a step-by-step task planning method, presenting one interface at a time to the LLM. The LLM then identifies relevant elements within the interface and determines the next action, enhancing accuracy and practicality. Extensive experiments show that VisionTasker outperforms previous methods, providing effective UI representations across four datasets. Additionally, in automating 147 real-world tasks on an Android smartphone, VisionTasker demonstrates advantages over humans in tasks where humans show unfamiliarity and shows significant improvements when integrated with the PBD mechanism. VisionTasker is open-source and available at https://github.com/AkimotoAyako/VisionTasker.
References in corpus (10)
- Learning Transferable Visual Models From Natural Language Supervision
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- PaLM: Scaling Language Modeling with Pathways
- Showing Academic Performance Predictions during Term Planning: Effects on Students' Decisions, Behaviors, and Preferences
- Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
- Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots
- AXNav: Replaying Accessibility Tests from Natural Language
- Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images
- Voicify Your UI: Towards Android App Control with Voice Commands
- Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus