Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment

Suqin Yuan, Runqi Lin, Muyang Li +5

Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not…