2 citations · 4 across the 11 of their papers we have counts for
11 papers
Dual-objective Language Models: Training Efficiency Without Overfitting
David Samuel, Lucas Georges Gabriel Charpentier
This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-ob…
HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
Stephan Oepen, Nikolay Arefev, Mikko Aulamo +29
We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely th…
Stronger Re-identification Attacks through Reasoning and Aggregation
Lucas Georges Gabriel Charpentier, Pierre Lison
Text de-identification techniques are often used to mask personally identifiable information (PII) from documents. Their ability to conceal the identity of the individuals mentione…
Systematic Generalization in Language Models Scales with Information Entropy
Sondre Wold, Lucas Georges Gabriel Charpentier, Étienne Simon
Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle wi…
Re-identification of De-identified Documents with Autoregressive Infilling
Lucas Georges Gabriel Charpentier, Pierre Lison
Documents revealing sensitive information about individuals must typically be de-identified. This de-identification is often done by masking all mentions of personally identifiable…
Small Languages, Big Models: A Study of Continual Training on Languages of Norway
David Samuel, Vladislav Mikhailov, Erik Velldal +4
Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages l…
GPT or BERT: why not both?
Lucas Georges Gabriel Charpentier, David Samuel
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both model…
Compositional Generalization with Grounded Language Models
Sondre Wold, Étienne Simon, Lucas Georges Gabriel Charpentier +3
Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous w…
More Room for Language: Investigating the Effect of Retrieval on Language Models
David Samuel, Lucas Georges Gabriel Charpentier, Sondre Wold
Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually re…
Not all layers are equally as important: Every Layer Counts BERT
Lucas Georges Gabriel Charpentier, David Samuel
This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participatin…
BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
Lucas Georges Gabriel Charpentier, Sondre Wold, David Samuel +1
Retrieval-based language models are increasingly employed in question-answering tasks. These models search in a corpus of documents for relevant information instead of having all f…