Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach

Xinnan Zhang, Chenliang Li, Siliang Zeng +6

Aligning large language models (LLMs) with human preferences usually requires fine-tuning methods such as RLHF and DPO. These methods directly optimize the model parameters, so the…