Towards Reliable, Uncertainty-Aware Alignment

Debangshu Banerjee, Kintan Saha, Aditya Gopalan

Alignment of large language models (LLMs) typically involves training a reward model on preference data, followed by policy optimization with respect to the reward model. However,…