Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

Yansen Han, Shengyi Liao, Yuanxing Zhang +2

Preference optimization is a standard alignment method for generative models, yet extending it to continuous-time dynamics remains non-trivial. In flow matching, reward-driven upda…