4 papers
TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization
Lingling Fu, Yongfu Xu
Direct Preference Optimization (DPO) has been widely adopted for large language model alignment due to its simple training procedure and lack of an explicit reward model. However,…
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Lingling Fu, Yongfu Xue
Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing rewa…
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
Yongfu Xue
Reward models are pivotal for aligning Large Language Models (LLMs) with human preferences. Existing approaches face two key limitations: Discriminative reward models require large…
Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
Yongfu Xue
The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread p…