8 papers · 1 filter
Functional Welfare for AI Safety
Rikhil Jhaveri, Jamie Johnson, Punya Syon Pandey +1
AI welfare is generally treated purely as an ethical issue rather than also as a safety issue. We argue this is a mistake. Whatever their moral status, models exhibit welfare-like…
Character Training for Risk-Averse Agents
Arav Dhoot, Punya Syon Pandey, Jamie Johnson +3
Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making dea…
PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner +17
Expert writing feedback from experienced researchers is critical for early-career scholars to improve their manuscripts, yet high-quality feedback often remains scarce because revi…
CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs
Florent Draye, Vedant Palit, Abir Harrasse +8
Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders…
Do LLMs Rewrite History? Detecting Historical Revisionism in Large Language Models
Francesco Ortu, Joeun Yook, Punya Syon Pandey +5
Large language models (LLMs) are increasingly used to disseminate information, yet their handling of contested historical events and politically charged narratives remains understu…
BinaryPPO: Efficient Policy Optimization for Binary Classification
Punya Syon Pandey, Zhijing Jin
Supervised fine-tuning (SFT) is the standard approach for binary classification tasks such as toxicity detection, factuality verification, and causal inference. However, SFT often…
TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering
Saad Hossain, Tom Tseng, Punya Syon Pandey +8
As increasingly capable open-weight large language models (LLMs) are deployed, improving their tamper resistance against unsafe modifications, whether accidental or intentional, be…
Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift
Daniel Vennemeyer, Punya Syon Pandey, Phan Anh Duong +2
Fine-tuning LLMs on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes r…