PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

Doo Hwan Hwang, Kee-Eung Kim

Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training. Despite…