2 citations · 4 across the 11 of their papers we have counts for
4 papers · 1 filter
Small Languages, Big Models: A Study of Continual Training on Languages of Norway
David Samuel, Vladislav Mikhailov, Erik Velldal +4
Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages l…
GPT or BERT: why not both?
Lucas Georges Gabriel Charpentier, David Samuel
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both model…
Compositional Generalization with Grounded Language Models
Sondre Wold, Étienne Simon, Lucas Georges Gabriel Charpentier +3
Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous w…
More Room for Language: Investigating the Effect of Retrieval on Language Models
David Samuel, Lucas Georges Gabriel Charpentier, Sondre Wold
Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually re…