Are PPO-ed Language Models Hackable?

Suraj Anand, David Getzen

Numerous algorithms have been proposed to $\text{align}$ language models to remove undesirable behaviors. However, the challenges associated with a very large state space and cre…