5 citations · 5 across the 22 of their papers we have counts for
23 papers
MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning
Changyue Jiang, Jiayi Wang, Xin Wen +3
Mobile agents powered by foundation models now automate complex, multi-step workflows on real devices, but their trajectories can violate app-specific security policies. Existing t…
Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability
Pei Chen, Baichao An, Mengying Wu +6
The Model Context Protocol (MCP) has rapidly established itself as a standard interface for enabling LLM-based agents to interact with external tools and services. As MCP servers a…
The 2026 Singapore Consensus on Global AI Safety Research Priorities
Stephen Casper, Oskar Galeev, Yoshua Bengio +117
Frontier AI capabilities and autonomy are advancing rapidly. A growing number of real-world incidents make a trusted AI ecosystem essential to embracing AI with confidence. The 202…
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
Jiaqi Luo, Jiarun Dai, Zhile Chen +8
Nowadays, the autonomous execution of cyberattacks capable of causing substantial real-world harm is widely regarded as one of the critical red lines that frontier AI systems must…
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
Fengyu Liu, Jiarun Dai, Yihe Fan +11
Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation. However, evaluating their offensive c…
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
Yihe Fan, Changyi Li, Lichen Xu +4
LLM-based agents are increasingly used for cybersecurity tasks, but most existing systems rely on fixed, human-designed scaffolds that struggle to adapt across diverse targets and…
AgentGuard: An Attribute-Based Access Control Framework for Tool-Use LLM-Based Agent
Jiaqi Luo, Songyang Peng, Jiarun Dai +6
LLM-based agents have recently attracted significant attention due to their ability to autonomously invoke relevant tools to accomplish complex tasks. However, recent studies have…
FlowGuard: Towards Lightweight In-Generation Safety Detection for Diffusion Models via Linear Latent Decoding
Jinghan Yang, Yihe Fan, Xudong Pan +1
Diffusion-based image generation models have advanced rapidly but pose a safety risk due to their potential to generate Not-Safe-For-Work (NSFW) content. Existing NSFW detection me…
Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Search
Yulin Shen, Xudong Pan, Geng Hong +1
Recent advances in the Model Context Protocol (MCP) have enabled large language models (LLMs) to invoke external tools with unprecedented ease. This creates a new class of powerful…
AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
Changyi Li, Pengfei Lu, Xudong Pan +2
As Large Language Models (LLMs) evolve into autonomous agents, existing safety evaluations face a fundamental trade-off: manual benchmarks are costly, while LLM-based simulators ar…
MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction
Wenqi Zhang, Yulin Shen, Changyue Jiang +3
Large foundation models are integrated into Computer Use Agents (CUAs), enabling autonomous interaction with operating systems through graphical user interfaces (GUIs) to perform c…
WebTrap Park: An Automated Platform for Systematic Security Evaluation of Web Agents
Xinyi Wu, Jiagui Chen, Geng Hong +4
Web Agents are increasingly deployed to perform complex tasks in real web environments, yet their security evaluation remains fragmented and difficult to standardize. We present We…
When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent
Xinyi Wu, Geng Hong, Yueyue Chen +5
Web agents, powered by large language models (LLMs), are increasingly deployed to automate complex web interactions. The rise of open-source frameworks (e.g., Browser Use, Skyvern-…
MCPZoo: A Large-Scale Dataset of Runnable Model Context Protocol Servers for AI Agent
Mengying Wu, Pei Chen, Geng Hong +6
Model Context Protocol (MCP) enables agents to interact with external tools, yet empirical research on MCP is hindered by the lack of large-scale, accessible datasets. We present M…
Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing
Wuyuao Mai, Geng Hong, Qi Liu +5
Penetration testing is critical for identifying and mitigating security vulnerabilities, yet traditional approaches remain expensive, time-consuming, and dependent on expert human…
PRISON: Unmasking the Criminal Potential of Large Language Models
Xinyi Wu, Geng Hong, Pei Chen +3
As large language models (LLMs) advance, concerns about their misconduct in complex social contexts intensify. Existing research overlooked the systematic understanding and assessm…
Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems
Yihe Fan, Wenqi Zhang, Xudong Pan +1
As foundation models grow increasingly more intelligent, reliable and trustworthy safety evaluation becomes more indispensable than ever. However, an important question arises: Whe…
ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
Changyi Li, Jiayi Wang, Xudong Pan +2
Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving f…
Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
Changyue Jiang, Wenqi Zhang, Xudong Pan +2
LLM-based agents solve complex tasks through iterative reasoning, tool use, and environment interaction, where each intermediate thought directly shapes subsequent actions. Small d…
Bare Minimum Mitigations for Autonomous AI Development
Joshua Clymer, Isabella Duan, Chris Cundy +10
Artificial intelligence (AI) is advancing rapidly, with the potential for significantly automating AI research and development itself in the near future. In 2024, international sci…
OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation
Yichen Wu, Qianqian Gao, Xudong Pan +2
As large language models (LLMs) are increasingly deployed as interactive agents, open-ended human-AI interactions can involve deceptive behaviors with serious real-world consequenc…
You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
Wuyuao Mai, Geng Hong, Pei Chen +5
With the rise of generative large language models (LLMs) like LLaMA and ChatGPT, these models have significantly transformed daily life and work by providing advanced insights. How…
Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications
Changyue Jiang, Xudong Pan, Geng Hong +3
Retrieval-Augmented Generation (RAG) expands the knowledge boundary of large language models (LLMs) by integrating external knowledge bases, whose construction is often time-consum…