4 papers
DnA: Denoising Attention for Visual Tasks
Ron Campos, Subhajit Maity, Xin Li +2
The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy atte…
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
Elijah Cadenhead, Cristian McGee, Xin Li +2
Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models. However, questions remain about the compa…
Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?
Subhajit Maity, Killian Hitsman, Xin Li +1
Kolmogorov-Arnold networks (KANs) are a remarkable innovation that consists of learnable activation functions, with the potential to capture more complex relationships from data. P…
Where Have All the Kaczmarz Iterates Gone?
El Houcine Bergou, Soumia Boucherouite, Aritra Dutta +2
The randomized Kaczmarz (RK) algorithm is one of the most computationally and memory-efficient iterative algorithms for solving large-scale linear systems. However, practical appli…