Mitigating the Alignment Tax of RLHF

Yong Lin, Hangyu Lin, Wei Xiong +14

LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, w…