75 citations · 130 across the 11 of their papers we have counts for
3 papers · 1 filter
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Maria Lomeli, Antoine Groudiev, Matthijs Douze +7
This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…
Cluster with Auctions for Vector Search
Swann Bessa, Pierre Fernandez, Gergely Szilvasy +2
Large-scale approximate nearest neighbor search commonly relies on partitions for indexing: database vectors are partitioned into clusters, and for each query a probing function se…
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-cont…