From the 1 of 19 linked papers with an AI index.
12 papers · 1 filter
Memory Efficient Tabular Foundation Models
Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2
The paper studies how to reduce the memory footprint of tabular foundation models like TabPFN using compression techniques, achieving up to 7.6× memory savings with little performa…
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran +1
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…
The Quantization Benefits of Residual-Free Transformers
Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantizati…
Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
Arpit Garg, Hemanth Saratchandran, Ravi Garg +1
Machine unlearning in foundation models (e.g., language and vision transformers) is essential for privacy and safety; however, existing approaches are unstable and unreliable. A wi…
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji +2
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax…
Spectral Conditioning of Attention Improves Transformer Performance
Hemanth Saratchandran, Simon Lucey
We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the a…