1 citations · 2 across the 11 of their papers we have counts for
6 papers · 1 filter
AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
Ruibo Deng, Duanyu Feng, Wenqiang Lei
Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models. However, their effectiveness is critically dependent on ranking a…
Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts
Youcheng Huang, Chen Huang, Duanyu Feng +2
Understanding the inner workings of Large Language Models (LLMs) is a critical research frontier. Prior research has shown that a single LLM's concept representations can be captur…
Dishonesty in Helpful and Harmless Alignment
Youcheng Huang, Jingkun Tang, Duanyu Feng +4
People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference.…
Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
Duanyu Feng, Bowen Qin, Chen Huang +3
The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture t…
Towards Understanding the Influence of Reward Margin on Preference Model Performance
Bowen Qin, Duanyu Feng, Xi Yang
Reinforcement Learning from Human Feedback (RLHF) is a widely used framework for the training of language models. However, the process of using RLHF to develop a language model tha…
Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
Duanyu Feng, Bowen Qin, Chen Huang +2
Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with…