1 citations · 1 across the 18 of their papers we have counts for
5 papers · 1 filter
AdaSplash-2: Faster Differentiable Sparse Attention
Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4
Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is -entmax a…
Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
Sergey Troshin, Wafaa Mohammed, Yan Meng +3
Diversity is an essential metric for evaluating the creativity of outputs generated by language models. Temperature-based sampling is a common strategy to increase diversity. Howev…
Keep your distance: learning dispersed embeddings on
Evgeniia Tokarchuk, Hua Chang Bakker, Vlad Niculae
Learning well-separated features in high-dimensional spaces, such as text or image embeddings, is crucial for many machine learning applications. Achieving such separation can be e…
Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval
Saul Santos, Vlad Niculae, Daniel McNamee +1
Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-atte…
Sparse and Structured Hopfield Networks
Saul Santos, Vlad Niculae, Daniel McNamee +1
Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by e…