4 citations · 4 across the 3 of their papers we have counts for
3 papers
Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
Nathan Godey, Éric de la Clergerie, Benoît Sagot
Recent advances in language modeling consist in pretraining highly parameterized neural networks on extremely large web-mined text corpora. Training and inference with such models…
On the Scaling Laws of Geographical Representation in Language Models
Nathan Godey, Éric de la Clergerie, Benoît Sagot
Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited by extending this result to La…
CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data
Rian Touchent, Laurent Romary, Eric de la Clergerie
Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However these documents are unstructured and it is therefore necessary to extr…