11 papers
Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training
Zheng-Xin Yong, Stephen H. Bach
We discover a novel and surprising phenomenon of unintentional misalignment in reasoning language models (RLMs), which we call self-jailbreaking. Specifically, after benign reasoni…
TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
Reza Esfandiarpoor, Vishwas Suryanarayanan, Stephen H. Bach +2
Since the introduction of the Model Context Protocol (MCP), the number of available tools for Large Language Models (LLMs) has increased significantly. These task-specific tool set…
Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance
Reza Esfandiarpoor, George Zerveas, Ruochen Zhang +3
Although synthetic data has changed various aspects of information retrieval (IR) pipelines, the main training paradigm remains: contrastive learning with binary relevance labels,…
Trove: A Flexible Toolkit for Dense Retrieval
Reza Esfandiarpoor, Max Zuo, Stephen H. Bach
We introduce Trove, an easy-to-use open-source retrieval toolkit that simplifies research experiments without sacrificing flexibility or speed. For the first time, we introduce eff…
Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models
Yik Siu Chan, Zheng-Xin Yong, Stephen H. Bach
Reasoning language models improve performance on complex tasks by generating long chains of thought (CoTs), but this process can also increase harmful outputs in adversarial settin…
The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee +2
This paper presents a comprehensive analysis of the linguistic diversity of LLM safety research, highlighting the English-centric nature of the field. Through a systematic review o…