6 papers
Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks
Yunzhe Xu, Zhuosheng Zhang, Zhe Liu
While prompt optimization has emerged as a critical technique for enhancing language model performance, existing approaches primarily focus on elicitation-based strategies that sea…
OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows
Qiushi Sun, Mukai Li, Zhoumianze Liu +11
Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms. While these agents…
HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization
Yurun Chen, Xavier Hu, Yuhan Liu +4
Large language models enable agents to autonomously perform tasks in open web environments. However, as hidden threats within the web evolve, web agents face the challenge of balan…
XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level
Shaoqing Zhang, Kehai Chen, Zhuosheng Zhang +4
Recent advancements in vision-language models have increased interest in Device-Control Agents (DC agents) for managing graphical user interfaces (GUIs). With the growing complexit…
Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM
Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen +4
Despite being empowered with alignment mechanisms, large language models (LLMs) are increasingly vulnerable to emerging jailbreak attacks that can compromise their alignment mechan…
Dynamic Planning for LLM-based Graphical User Interface Automation
Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen +4
The advent of large language models (LLMs) has spurred considerable interest in advancing autonomous LLMs-based agents, particularly in intriguing applications within smartphone gr…