75 citations · 130 across the 11 of their papers we have counts for
12 papers
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Maria Lomeli, Antoine Groudiev, Matthijs Douze +7
This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…
Cluster with Auctions for Vector Search
Swann Bessa, Pierre Fernandez, Gergely Szilvasy +2
Large-scale approximate nearest neighbor search commonly relies on partitions for indexing: database vectors are partitioned into clusters, and for each query a probing function se…
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-cont…
Short window attention enables long-term memorization
Loïc Cabannes, Maximilian Beck, Gergely Szilvasy +6
Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. H…
Stochastic activations
Maria Lomeli, Matthijs Douze, Gergely Szilvasy +7
We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, w…
DINOv3
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer +23
Self-supervised learning holds the promise of eliminating the need for manual data annotation, enabling models to scale effortlessly to massive datasets and larger architectures. B…
Inference-time sparse attention with asymmetric indexing
Pierre-Emmanuel Mazaré, Gergely Szilvasy, Maria Lomeli +4
Self-attention in transformer models is an incremental associative memory that maps key vectors to value vectors. One way to speed up self-attention is to employ GPU-compatible vec…
Moshi: a speech-text foundation model for real-time dialogue
Alexandre Défossez, Laurent Mazaré, Manu Orsini +5
We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namel…
Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach
Huy V. Vo, Vasil Khalidov, Timothée Darcet +10
Self-supervised features are the cornerstone of modern machine learning systems. They are typically pre-trained on data collections whose construction and curation typically requir…
The Faiss library
Matthijs Douze, Alexandr Guzhva, Chengqi Deng +6
Vector databases typically manage large collections of embedding vectors. Currently, AI applications are growing rapidly, and so is the number of embeddings that need to be stored…
The Stable Signature: Rooting Watermarks in Latent Diffusion Models
Pierre Fernandez, Guillaume Couairon, Hervé Jégou +2
Generative image modeling enables a wide range of applications but raises ethical concerns about responsible deployment. This paper introduces an active strategy combining image wa…
Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models
Matthew J. Muckley, Alaaeldin El-Nouby, Karen Ullrich +2
Lossy image compression aims to represent images in as few bits as possible while maintaining fidelity to the original. Theoretical results indicate that optimizing distortion metr…