InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

Yu Li, Tian Lan, Zhengling Qi

Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify tw…