752 citations · 836 across the 11 of their papers we have counts for
13 papers · 1 filter
Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
Nathan Godey, Alessio Devoto, Yu Zhao +4
Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths…
CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
Wissam Antoun, Francis Kulumba, Rian Touchent +3
French language models, such as CamemBERT, have been widely adopted across industries for natural language processing (NLP) tasks, with models like CamemBERT seeing over 4 million…
Anisotropy Is Inherent to Self-Attention in Transformers
Nathan Godey, Éric de la Clergerie, Benoît Sagot
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotrop…
Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Nathan Godey, Éric de la Clergerie, Benoît Sagot
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative…
Is Anisotropy Inherent to Transformers?
Nathan Godey, Éric de la Clergerie, Benoît Sagot
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotrop…
MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling
Nathan Godey, Roman Castagné, Éric de la Clergerie +1
Static subword tokenization algorithms have been an essential component of recent works on language modeling. However, their static nature results in important flaws that degrade t…
Rethinking Automatic Evaluation in Sentence Simplification
Thomas Scialom, Louis Martin, Jacopo Staiano +2
Automatic evaluation remains an open research question in Natural Language Generation. In the context of Sentence Simplification, this is particularly challenging: the task require…
Clustering-based Automatic Construction of Legal Entity Knowledge Base from Contracts
Fuqi Song, Éric de la Clergerie
In contract analysis and contract automation, a knowledge base (KB) of legal entities is fundamental for performing tasks such as contract verification, contract generation and con…
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases
Louis Martin, Angela Fan, Éric de la Clergerie +2
Progress in sentence simplification has been hindered by a lack of labeled parallel simplification data, particularly in languages other than English. We introduce MUSS, a Multilin…
CamemBERT: a Tasty French Language Model
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez +5
Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concat…
Controllable Sentence Simplification
Louis Martin, Benoît Sagot, Éric de la Clergerie +1
Text simplification aims at making a text easier to read and understand by simplifying grammar and structure while keeping the underlying information identical. It is often conside…
Reference-less Quality Estimation of Text Simplification Systems
Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré +3
The evaluation of text simplification (TS) systems remains an open challenge. As the task has common points with machine translation (MT), TS is often evaluated using MT metrics su…
Évaluation de lexiques syntaxiques par leur intégartion dans l'analyseur syntaxiques FRMG
Elsa Tolone, Éric De La Clergerie, Sagot Benoit
In this paper, we evaluate various French lexica with the parser FRMG: the Lefff, LGLex, the lexicon built from the tables of the French Lexicon-Grammar, the lexicon DICOVALENCE an…