On the Entropy Calibration of Language Models

Steven Cao, Gregory Valiant, Percy Liang

We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text. Past work found that models are misc…