Multi-dimensional Preference Alignment by Conditioning Reward Itself

Jiho Jang, Jinyoung Kim, Kyungjune Baek +1

Reinforcement Learning from Human Feedback has emerged as a standard for aligning diffusion models. However, we identify a fundamental limitation in the standard DPO formulation be…