From the 1 of 19 linked papers with an AI index.
19 papers
Mask-Based Priors Are More Persistent than Query-Key Initializations
Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1
Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that,…
Memory Efficient Tabular Foundation Models
Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2
The paper studies how to reduce the memory footprint of tabular foundation models like TabPFN using compression techniques, achieving up to 7.6× memory savings with little performa…
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran +1
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…
Parameter-Efficient Fine-Tuning with Learnable Rank
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bia…
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
Runze Xu, Arpit Garg, Hemanth Saratchandran +1
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation perform…
The Quantization Benefits of Residual-Free Transformers
Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantizati…