Procedural Pretraining: Warming Up Language Models with Abstract Data

Liangze Jiang, Zachary Shinnick, Anton van den Hengel +2

Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease…