From the 1 of 13 linked papers with an AI index.
6 papers · 1 filter
Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
Lina Berrayana, Ahmed Heakl, Muhammad Abdullah Sohail +3
Current autoregressive language models (ARMs) achieve high accuracy but require long token sequences, making them costly. Discrete diffusion language models (DDLMs) enable parallel…
Causal Estimation of Tokenisation Bias
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps…
Local and Global Decoding in Text Generation
Daniel Gareev, Thomas Hofmann, Ezhilmathi Krishnasamy +1
Text generation, a key component in applications such as dialogue systems, relies on decoding algorithms that sample strings from a language model distribution. Traditional methods…
The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments
Anton Schäfer, Shauli Ravfogel, Thomas Hofmann +2
Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple langu…
On the Effect of (Near) Duplicate Subwords in Language Modelling
Anton Schäfer, Thomas Hofmann, Imanol Schlag +1
Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned arbitrary indices before being served to the LM. W…
Explicit Word Density Estimation for Language Modelling
Jovan Andonov, Octavian Ganea, Paulina Grnarova +2
Language Modelling has been a central part of Natural Language Processing for a very long time and in the past few years LSTM-based language models have been the go-to method for c…