Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data

Shuai Zhao, Yunqiu Xu, Linchao Zhu +1

Large language models~(LLMs) are expected to be helpful, harmless, and honest. In different alignment scenarios, such as safety, confidence, and general preference alignment, binar…