4 papers
Attention to Mamba: A Recipe for Cross-Architecture Distillation
Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane +3
State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compare…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation
Skyler Seto, Barry-John Theobald, Federico Danieli +2
Fully-test-time adaptation (F-TTA) can mitigate performance loss due to distribution shifts between train and test data (1) without access to the training data, and (2) without kno…
Space-Time Block Preconditioning for Incompressible Resistive Magnetohydrodynamics
Federico Danieli, Ben S. Southworth, Jacob B. Schroder
This work develops an all-at-once space-time preconditioning approach for resistive magnetohydrodynamics (MHD). We consider parallel-in-time due to the long time domains required t…