13 papers
Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Junjie Chu, Xinyue Shen, Ye Leng +3
The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastructures for identifying key trends and fa…
Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models
Zeyuan Chen, Yihan Ma, Xinyue Shen +2
Large language models (LLMs) show strong performance across many applications, but their ability to memorize and potentially reveal training data raises serious privacy concerns. W…
Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
Xue Liu, Xin Ma, Yuxin Ma +36
As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks c…
HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
Yukun Jiang, Yage Zhang, Michael Backes +2
Large language models (LLMs) have evolved into autonomous agents that rely on open skill ecosystems (e.g., ClawHub and Skills.Rest), hosting numerous publicly reusable skills. Exis…
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
Rui Zhang, Hongwei Li, Yun Shen +6
The deployment of large language models (LLMs) raises significant ethical and safety concerns. While LLM alignment techniques are adopted to improve model safety and trustworthines…
OrgAgent: Organize Your Multi-Agent System like a Company
Yiru Wang, Xinyue Shen, Yaohui Han +3
While large language model-based multi-agent systems have shown strong potential for complex reasoning, how to effectively organize multiple agents remains an open question. In thi…