Likelihood Hacking in Probabilistic Program Synthesis

Jacek Karwowski, Younesse Kaddar, Zihuiwen Ye +2

When language models are trained by reinforcement learning (RL) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing program…