Do we really have to filter out random noise in pre-training data for language models?

Jinghan Ru, Yuxin Xie, Xianwei Zhuang +5

Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unr…