From the 1 of 15 linked papers with an AI index.
15 papers
Cluster with Auctions for Vector Search
Swann Bessa, Pierre Fernandez, Gergely Szilvasy +2
The paper introduces CwA, a method that jointly learns a balanced clustering of database vectors and a neural probing function for large‑scale vector search, using a parallelizable…
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-cont…
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5
Under modern test-time compute and agentic paradigms, language models process ever-longer sequences. Efficient text generation with transformer architectures is increasingly constr…
Short window attention enables long-term memorization
Loïc Cabannes, Maximilian Beck, Gergely Szilvasy +6
Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. H…
Stochastic activations
Maria Lomeli, Matthijs Douze, Gergely Szilvasy +7
We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, w…
The Faiss library
Matthijs Douze, Alexandr Guzhva, Chengqi Deng +6
Vector databases typically manage large collections of embedding vectors. Currently, AI applications are growing rapidly, and so is the number of embeddings that need to be stored…