2 papers
cs.IR2026
TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization
Lingling Fu, Yongfu Xu
Direct Preference Optimization (DPO) has been widely adopted for large language model alignment due to its simple training procedure and lack of an explicit reward model. However,…
cs.LG2026
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Lingling Fu, Yongfu Xue
Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing rewa…