1 citations · 1 across the 13 of their papers we have counts for
Showing 2023Show all
2 papers · 1 filter
cs.CL2023
Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Nathan Godey, Éric de la Clergerie, Benoît Sagot
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative…
cs.CL2023★ 1 cited
Is Anisotropy Inherent to Transformers?
Nathan Godey, Éric de la Clergerie, Benoît Sagot
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotrop…