4 papers
SkillHarness: Harnessing Safe Skills for Computer-Use Agents
Yurun Chen, Biao Yi, Keting Yin +1
Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches…
What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
Songze Li, Xiaoke Guo, Tianqi Liu +5
Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct screen-based decision-making,…
EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation
Biao Yi, Xavier Hu, Yurun Chen +3
To tackle increasingly complex tasks, recent research on mobile agents has shifted towards multi-agent collaboration. Current mobile multi-agent systems are primarily deployed in t…
OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
Xueyu Hu, Tao Xiong, Biao Yi +26
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of (multi-modal) large la…