Showing cs.CLShow all
2 papers · 1 filter
cs.CL2026
A Language Model from 1913: Pretraining on Historical Text
Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5
While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text) in a data-constrained setting can produ…
cs.CL2026
Procedural Pretraining: Warming Up Language Models with Abstract Data
Liangze Jiang, Zachary Shinnick, Anton van den Hengel +2
Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease…