Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity

Rheeya Uppaal, Apratim Dey, Yiting He +2

Recent alignment algorithms such as direct preference optimization (DPO) have been developed to improve the safety of large language models (LLMs) by training these models to match…