7 papers
Lasting Effects of Abstract Pretraining Beyond Perplexity
Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1
Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated dat…
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of a…
Procedural Pretraining for Molecular Property Prediction
Moritz Friedemann, Zachary Shinnick, Philip Torr +1
Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules. In this work, we as…
A Language Model from 1913: Pretraining on Historical Text
Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5
While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text) in a data-constrained setting can produ…
Procedural Pretraining: Warming Up Language Models with Abstract Data
Liangze Jiang, Zachary Shinnick, Anton van den Hengel +2
Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease…
Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Transformers are remarkably versatile, suggesting the existence of generic inductive biases beneficial across modalities. In this work, we explore a new way to instil such biases i…
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally thr…