461 citations · 674 across the 16 of their papers we have counts for
19 papers
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Maria Lomeli, Antoine Groudiev, Matthijs Douze +7
This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-cont…
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5
Under modern test-time compute and agentic paradigms, language models process ever-longer sequences. Efficient text generation with transformer architectures is increasingly constr…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Short window attention enables long-term memorization
Loïc Cabannes, Maximilian Beck, Gergely Szilvasy +6
Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. H…
Stochastic activations
Maria Lomeli, Matthijs Douze, Gergely Szilvasy +7
We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, w…
RTTC: Reward-Guided Collaborative Test-Time Compute
J. Pablo Muñoz, Jinjie Yuan
Test-Time Compute (TTC) has emerged as a powerful paradigm for enhancing the performance of Large Language Models (LLMs) at inference, leveraging strategies such as Test-Time Train…
Assay2Mol: large language model-based drug design using BioAssay context
Yifan Deng, Spencer S. Ericksen, Anthony Gitter
Scientific databases aggregate vast amounts of quantitative data alongside descriptive text. In biochemistry, molecule screening assays evaluate candidate molecules' functional res…
Inference-time sparse attention with asymmetric indexing
Pierre-Emmanuel Mazaré, Gergely Szilvasy, Maria Lomeli +4
Self-attention in transformer models is an incremental associative memory that maps key vectors to value vectors. One way to speed up self-attention is to employ GPU-compatible vec…
Vector search with small radiuses
Gergely Szilvasy, Pierre-Emmanuel Mazaré, Matthijs Douze
In recent years, the dominant accuracy metric for vector search is the recall of a result list of fixed size (top-k retrieval), considering as ground truth the exact vector retriev…
The Faiss library
Matthijs Douze, Alexandr Guzhva, Chengqi Deng +6
Vector databases typically manage large collections of embedding vectors. Currently, AI applications are growing rapidly, and so is the number of embeddings that need to be stored…
Improving Wikipedia Verifiability with AI
Fabio Petroni, Samuel Broscheit, Aleksandra Piktus +10
Verifiability is a core content policy of Wikipedia: claims that are likely to be challenged need to be backed by citations. There are millions of articles available online and tho…
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve +4
Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech…
Unsupervised pretraining transfers well across languages
Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré +1
Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting. This assumes the existence of a parallel…
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Jacob Kahn, Morgane Rivière, Weiyi Zheng +12
We introduce a new collection of spoken English audio suitable for training speech recognition systems under limited or no supervision. It is derived from open-source audio books f…
Reference-less Quality Estimation of Text Simplification Systems
Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré +3
The evaluation of text simplification (TS) systems remains an open challenge. As the task has common points with machine translation (MT), TS is often evaluated using MT metrics su…
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré +1
The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapp…
Training Millions of Personalized Dialogue Agents
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison +1
Current dialogue systems are not very engaging for users, especially when trained end-to-end without relying on proactive reengaging scripted strategies. Zhang et al. (2018) showed…
Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading
Martin Raison, Pierre-Emmanuel Mazaré, Rajarshi Das +1
This paper aims at improving how machines can answer questions directly from text, with the focus of having models that can answer correctly multiple types of questions and from va…