Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Minsu Kim, Jianxun Lian, Xing Xie +1

Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-train…