4 papers · 1 filter
AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor
Shu Yang, Jingyu Hu, Tong Li +3
We introduce AutoMonitor-Bench, the first benchmark designed to systematically evaluate the reliability of LLM-based misbehavior monitors across diverse tasks and failure modes. Au…
Faithful-Patchscopes: Understanding and Mitigating Model Bias in Hidden Representations Explanation of Large Language Models
Xilin Gong, Shu Yang, Zehua Cao +2
Large Language Models (LLMs) have demonstrated strong capabilities for hidden representation interpretation through Patchscopes, a framework that uses LLMs themselves to generate h…
Investigating CoT Monitorability in Large Reasoning Models
Shu Yang, Junchao Wu, Xilin Gong +4
Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex tasks by engaging in extended reasoning before producing final answers. Beyond improving abilities…
Benchmarking Language Model Creativity: A Case Study on Code Generation
Yining Lu, Dixuan Wang, Tianjian Li +4
As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key character…