Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

Abstract

UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28\% success rate in the baseline to 95.24\% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service.

34 pages

References in corpus (9)

• Survey of Hallucination in Natural Language Generation
• Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
• AXNav: Replaying Accessibility Tests from Natural Language
• HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
• Synapse: Interactive Guidance by Demonstration with Trial-and-Error Support for Older Adults to Use Smartphone Apps
• VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
• GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning
• Voicify Your UI: Towards Android App Control with Voice Commands
• Automatic Macro Mining from Interaction Traces at Scale