Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Xiang Ji, Sanjeev Kulkarni, Mengdi Wang +1

This work studies the challenge of aligning large language models (LLMs) with offline preference data. We focus on alignment by Reinforcement Learning from Human Feedback (RLHF) in…