1 citations · 2 across the 8 of their papers we have counts for
6 papers · 1 filter
Sparse Attention as Compact Kernel Regression
Saul Santos, Nuno Gonçalves, Daniel C. McNamee +2
Recent work has revealed a link between self-attention mechanisms in transformers and test-time kernel regression via the Nadaraya-Watson estimator, with standard softmax attention…
TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
André G. Viveiros, Patrick Fernandes, Saul Santos +7
Despite significant advances in vision-language models (VLMs), most existing work follows an English-centric design process, limiting their effectiveness in multilingual settings.…
Modern Hopfield Networks with Continuous-Time Memories
Saul Santos, António Farinhas, Daniel C. McNamee +1
Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these…
Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval
Saul Santos, Vlad Niculae, Daniel McNamee +1
Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-atte…
Sparse and Structured Hopfield Networks
Saul Santos, Vlad Niculae, Daniel McNamee +1
Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by e…
Symplectic Momentum Neural Networks -- Using Discrete Variational Mechanics as a prior in Deep Learning
Saul Santos, Monica Ekal, Rodrigo Ventura
With deep learning gaining attention from the research community for prediction and control of real physical systems, learning important representations is becoming now more than e…