2 papers
cs.IR2026
TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization
Lingling Fu, Yongfu Xu
Direct Preference Optimization (DPO) has been widely adopted for large language model alignment due to its simple training procedure and lack of an explicit reward model. However,…
cs.LG2026
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Lingling Fu, Yongfu Xue
Reward models (RMs) are a critical component of reinforcement learning from human feedback (RLHF). However, conventional dense RMs are susceptible to exploitation by policy models…