An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

Qizhen Zhang, Ankush Garg, Jakob Foerster +3

Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulat…