93 citations · 446 across the 32 of their papers we have counts for
3 papers · 1 filter
Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Transformers are remarkably versatile, suggesting the existence of generic inductive biases beneficial across modalities. In this work, we explore a new way to instil such biases i…
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally thr…
Leaner Transformers: More Heads, Less Depth
Hemanth Saratchandran, Damien Teney, Simon Lucey
Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. Thi…