From Distributional to Overton Pluralism: Investigating Large Language Model Alignment

Thom Lake, Eunsol Choi, Greg Durrett

The alignment process changes several properties of a large language model's (LLM's) output distribution. We analyze two aspects of post-alignment distributional shift of LLM respo…