A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

Indranil Halder, Rastri Dey, Cengiz Pehlevan

Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic que…