4 papers
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Hongxin Li, Xiping Wang, Jingran Su +4
Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends…
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Hongxin Li, Yuntao Chen, Zhaoxiang Zhang
Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs…
UIPro: Unleashing Superior Interaction Capability For GUI Agents
Hongxin Li, Jingran Su, Jingfan Chen +4
Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence. Central to these a…
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Hongxin Li, Jingfan Chen, Jingran Su +3
User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation. However, existing datasets used…