1 citations · 3 across the 11 of their papers we have counts for
9 papers · 1 filter
Interpreting Model Reasoning through Behavioral Counterfactuals
Shawn Im, James Oldfield, Adel Bibi +1
Faithfully understanding the internal process that occurs as models form their chain of thought is an important problem with the rise in model capabilities. However, this remains c…
How Much Do Activation Monitors Benefit from Original Model Computations?
James Oldfield, Shawn Im, Jay J Culligan +3
One powerful way to safeguard AI systems is to monitor their internal computations--where signs of undesirable behavior might be more easily detectable than from models' inputs and…
Normalized Rewards for Preference Optimization
Shawn Im, Federico Danieli, Skyler Seto +2
Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their…
Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learning
Wendi Li, Shawn Im, Sharon Li
Agentic large language models are increasingly used to solve real-world tasks by reasoning over goals, invoking tools, and interacting with external environments. Reinforcement lea…
How Well Can Preference Optimization Generalize Under Noisy Feedback?
Shawn Im, Sharon Li
As large language models (LLMs) advance their capabilities, aligning these models with human preferences has become crucial. Preference optimization, which trains models to disting…
Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders
James Oldfield, Shawn Im, Sharon Li +3
Multilayer perceptrons (MLPs) are an integral part of large language models, yet their dense representations render them difficult to understand, edit, and steer. Recent methods le…
A Unified Understanding and Evaluation of Steering Methods
Shawn Im, Sharon Li
Latent space steering methods provide a practical approach to controlling large language models by applying steering vectors to intermediate activations, guiding outputs toward des…
Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
Shawn Im, Sharon Li
Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference lea…
Understanding the Learning Dynamics of Alignment with Human Feedback
Shawn Im, Yixuan Li
Aligning large language models (LLMs) with human intentions has become a critical task for safely deploying models in real-world systems. While existing alignment approaches have s…