472 citations · 1.5k across the 53 of their papers we have counts for
61 papers
RaBitQ-SSD: Split Codes and Pipelined I/O for SSD-Resident Vector Search
Yuexuan Xu, Jianyang Gao, Michael Norris +5
SSD-resident approximate nearest-neighbor search is essential when vector collections exceed DRAM capacity. The challenge is to reduce SSD reads and hide I/O latency through concur…
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Maria Lomeli, Antoine Groudiev, Matthijs Douze +7
This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…
Cluster with Auctions for Vector Search
Swann Bessa, Pierre Fernandez, Gergely Szilvasy +2
Large-scale approximate nearest neighbor search commonly relies on partitions for indexing: database vectors are partitioned into clusters, and for each query a probing function se…
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-cont…
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5
Under modern test-time compute and agentic paradigms, language models process ever-longer sequences. Efficient text generation with transformer architectures is increasingly constr…
Short window attention enables long-term memorization
Loïc Cabannes, Maximilian Beck, Gergely Szilvasy +6
Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. H…