Web Crawler Restrictions, AI Training Datasets \& Political Biases

Paul Bouchaud, Pedro Ramaciotti

Large language models rely on web-scraped text for training; concurrently, content creators are increasingly blocking AI crawlers to retain control over their data. We analyze craw…