KL-Regularized Reinforcement Learning is Designed to Mode Collapse

Anthony GX-Chen, Jatin Prakash, Jeff Guo +2

It is commonly believed that optimizing the reverse KL divergence results in "mode seeking", while optimizing forward KL results in "mass covering", with the latter being preferred…