6 citations · 11 across the 19 of their papers we have counts for
22 papers
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Bangya Liu, Jianhong Tu, Yueyang Pan +18
AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against…
AgentXploit: Autonomous Repository-to-Runtime Red-Teaming for AI Agents
Weida Liang, Shi Qiu, Zhun Wang +6
AI agents combine language models with external data and tools that can modify files, call APIs, or execute code. Security failures can arise when adversarial content changes an ag…
CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
Tianneng Shi, Robin Rheem, Dongwei Jiang +13
AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity e…
Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens
Peizhi Niu, Wenjie Qu, Shangding Gu +14
Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services. They take on system-level responsibilities…
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Xiaoyuan Liu, Jianhong Tu, Yuqi Chen +26
Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, cr…
Agents' Last Exam
Yiyou Sun, Xinyang Han, Weichen Zhang +378
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional d…
DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
Zhaorun Chen, Xun Liu, Haibo Tong +14
AI agents are increasingly deployed across diverse domains to automate complex workflows through long-horizon and high-stakes action executions. Due to their high capability and fl…
Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents
Tianci Xue, Zeyi Liao, Tianneng Shi +5
Real-world digital environments are highly diverse and dynamic. These characteristics cause agents to frequently encounter unseen environments and distribution shifts, making conti…
DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
Yuheng Tang, Kaijie Zhu, Bonan Ruan +14
Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown. D…
FrontierCS: Evolving Challenges for Evolving Intelligence
Qiuyang Mang, Wenhao Chai, Zhifei Li +48
We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competiti…
Measuring Agents in Production
Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo +22
LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first syst…
PromptArmor: Simple yet Effective Prompt Injection Defenses
Tianneng Shi, Kaijie Zhu, Zhun Wang +13
Despite their potential, recent research has demonstrated that LLM agents are vulnerable to prompt injection attacks, where malicious prompts are injected into the agent's input, c…
AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
Zhun Wang, Vincent Siu, Zhe Ye +6
The strong planning and reasoning capabilities of Large Language Models (LLMs) have fostered the development of agent-based systems capable of leveraging external tools and interac…
CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
Zhun Wang, Tianneng Shi, Jingxuan He +3
AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical. However, existing evaluations fall short, because they a…
Progent: Securing AI Agents with Privilege Control
Tianneng Shi, Jingxuan He, Zhun Wang +4
AI agents interact with external environments through tool calls, exposing them to attacks like indirect prompt injection that can trigger unauthorized actions. Securing these agen…
Frontier AI's Impact on the Cybersecurity Landscape
Yujin Potter, Wenbo Guo, Zhun Wang +6
The impact of frontier AI (i.e., AI agents and foundation models) in cybersecurity is rapidly increasing. In this paper, we comprehensively analyze this trend through multiple aspe…
Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs
Will Cai, Tianneng Shi, Xuandong Zhao +1
Commercial Large Language Model (LLM) APIs create a fundamental trust problem: users pay for specific models but have no guarantee that providers deliver them faithfully. Providers…
An Illusion of Progress? Assessing the Current State of Web Agents
Tianci Xue, Weijian Qi, Tianneng Shi +5
As digitalization and cloud technologies evolve, the web is becoming increasingly important in the modern society. Autonomous web agents based on large language models (LLMs) hold…
Improving LLM Safety Alignment with Dual-Objective Optimization
Xuandong Zhao, Will Cai, Tianneng Shi +4
Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed…
DeServe: Towards Affordable Offline LLM Inference via Decentralization
Linyu Wu, Xiaoyuan Liu, Tianneng Shi +2
The rapid growth of generative AI and its integration into everyday workflows have significantly increased the demand for large language model (LLM) inference services. While propr…
Can LLMs Ask Good Questions?
Yueheng Zhang, Xiaoyuan Liu, Yiyou Sun +5
We evaluate questions generated by large language models (LLMs) from context, comparing them to human-authored questions across six dimensions: question type, question length, cont…
UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworks
Xiaoyuan Liu, Tianneng Shi, Chulin Xie +9
Federated Learning (FL) has become a practical and widely adopted distributed learning paradigm. However, the lack of a comprehensive and standardized solution covering diverse use…