ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining

Zichun Yu, Jiarui Yan, Shlok Sanghvi +2

LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then c…