Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

Sofiia Riazhskykh, Nam Luu, Ondřej Bojar

Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33% of training tokens needed t…