13 papers
Task-Adaptive Rubrics for GUI Reward Modeling
Tao Xiong, Xavier Hu, Wenkai Wang +6
Recent studies on GUI agents have increasingly focused on outcome reward modeling, which assigns outcome rewards by judging whether an executed trajectory satisfies the success cri…
DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration
Wenkai Wang, Tao Xiong, Jingchen Ni +6
Real-world professional desktop workflows in specialized creative and engineering software unfold over long horizons and often require human-in-the-loop coordination, where agents…
Learning to Adapt: Self-Improving Web Agent via Cognitive-Aware Exploration
Weile Chen, Bingchen Miao, Qifan Yu +6
Recent advances in Multimodal Large Language Models (MLLMs) have led to promising progress in web agents. However, existing web agents often rely on handcrafted execution pipelines…
MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search
Sheng Zhang, Junyi Li, Yingyi Zhang +7
Recent advances in large language models (LLMs) have scaled the potential for reasoning and agentic search, wherein models autonomously plan, retrieve, and reason over external kno…
SafePred: A Predictive Guardrail for Computer-Using Agents via World Models
Yurun Chen, Zeyi Liao, Ping Yin +3
With the widespread deployment of Computer-using Agents (CUAs) in complex real-world environments, prevalent long-term risks often lead to severe and irreversible consequences. Mos…
GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection
Zishu Wei, Qixiang Ma, Xavier Hu +6
Building AI systems for GUI automation task has attracted remarkable research efforts, where MLLMs are leveraged for processing user requirements and give operations. However, GUI…
Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents
Reuben Tan, Baolin Peng, Zhengyuan Yang +15
Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-bas…
Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs
Yurun Chen, Xavier Hu, Yuhan Liu +8
As multimodal LLM-driven agents advance in autonomy and generalization, traditional static datasets face inherent scalability limitations and are insufficient for fully assessing t…
GUI-PRA: Process Reward Agent for GUI Tasks
Tao Xiong, Xavier Hu, Yurun Chen +6
Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide dense step-level supervision for…
OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
Xueyu Hu, Tao Xiong, Biao Yi +26
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of (multi-modal) large la…
HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization
Yurun Chen, Xavier Hu, Yuhan Liu +4
Large language models enable agents to autonomously perform tasks in open web environments. However, as hidden threats within the web evolve, web agents face the challenge of balan…
Efficient Agents: Building Effective Agents While Reducing Cost
Ningning Wang, Xavier Hu, Pai Liu +11
The remarkable capabilities of Large Language Model (LLM)-driven agents have enabled sophisticated systems to tackle complex, multi-step tasks, but their escalating costs threaten…
Mixture of Reasonings: Teach Large Language Models to Reason with Adaptive Strategies
Tao Xiong, Xavier Hu, Wenyan Fan +1
Large language models (LLMs) excel in complex tasks through advanced prompting techniques like Chain-of-Thought (CoT) and Tree-of-Thought (ToT), but their reliance on manually craf…