3 citations · 3 across the 11 of their papers we have counts for
6 papers · 1 filter
What Language is This? Ask Your Tokenizer
Clara Meister, Ahmetcan Yavuz, Pietro Lesci +1
Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, an…
Convergence and Divergence of Language Models under Different Random Seeds
Finlay Fehlauer, Kyle Mahowald, Tiago Pimentel
In this paper, we investigate the convergence of language models (LMs) trained under different random seeds, measuring convergence as the expected per-token Kullback--Leibler (KL)…
Causal Estimation of Tokenisation Bias
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps…
Local and Global Decoding in Text Generation
Daniel Gareev, Thomas Hofmann, Ezhilmathi Krishnasamy +1
Text generation, a key component in applications such as dialogue systems, relies on decoding algorithms that sample strings from a language model distribution. Traditional methods…
The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments
Anton Schäfer, Shauli Ravfogel, Thomas Hofmann +2
Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple langu…
On the Effect of (Near) Duplicate Subwords in Language Modelling
Anton Schäfer, Thomas Hofmann, Imanol Schlag +1
Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned arbitrary indices before being served to the LM. W…