Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization

Tina Behnia, Puneesh Deora, Christos Thrampoulidis

Language models are pretrained on sequences that blend statistical regularities (making text fluent) with factual associations between specific tokens (knowledge of facts). While r…