7 papers
UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations
Zihan Ding, Longxu Dou, Qi Gao +26
Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workfl…
Demo2Tutorial: From Human Experience to Multimodal Software Tutorials
Zechen Bai, Zhiheng Chen, Yiqi Lin +5
Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutori…
CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
Haobo Hu, Xiangwu Guo, Zhiheng Chen +4
While GUI agents have made significant progress in web navigation and basic operating system tasks, their capabilities in professional creative workflows remain largely underexplor…
ShowUI-Aloha: Human-Taught GUI Agent
Yichun Zhang, Xiangwu Guo, Yauhong Goh +5
Graphical User Interfaces (GUIs) are central to human-computer interaction, yet automating complex GUI tasks remains a major challenge for autonomous agents, largely due to a lack…
AUTO-Explorer: Automated Data Collection for GUI Agent
Xiangwu Guo, Difei Gao, Mike Zheng Shou
Recent advancements in GUI agents have significantly expanded their ability to interpret natural language commands to manage software interfaces. However, acquiring GUI data remain…
GUI Action Narrator: Where and When Did That Action Take Place?
Qinchen Wu, Difei Gao, Kevin Qinghong Lin +6
The advent of Multimodal LLMs has significantly enhanced image OCR recognition capabilities, making GUI automation a viable reality for increasing efficiency in digital tasks. One…
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Difei Gao, Lei Ji, Zechen Bai +10
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Lan…