4 papers
GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection
Zishu Wei, Qixiang Ma, Xavier Hu +6
Building AI systems for GUI automation task has attracted remarkable research efforts, where MLLMs are leveraged for processing user requirements and give operations. However, GUI…
OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
Xueyu Hu, Tao Xiong, Biao Yi +26
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of (multi-modal) large la…
Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration
Zheqi Lv, Keming Ye, Zishu Wei +7
Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes…
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Yuhang Liu, Pengxiang Li, Zishu Wei +7
Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers an…