7 citations · 22 across the 46 of their papers we have counts for
56 papers
Lasting Effects of Abstract Pretraining Beyond Perplexity
Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1
Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated dat…
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of a…
Conditioned Initialization for Attention
Hemanth Saratchandran, Simon Lucey
Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their success lies the attention laye…
Making Transformers Logical
Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1
Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that,…
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran +1
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…
Parameter-Efficient Fine-Tuning with Learnable Rank
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bia…