When Bad Data Leads to Good Models

Kenneth Li, Yida Chen, Fernanda Viégas +1

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- an…