10 citations · 10 across the 9 of their papers we have counts for
4 papers · 1 filter
From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes
Zébulon Goriely, Richard Diehl Martinez, Andrew Caines +2
Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes ca…
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
Richard Diehl Martinez, Zebulon Goriely, Andrew Caines +2
Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize…
Tending Towards Stability: Convergence Challenges in Small Language Models
Richard Diehl Martinez, Pietro Lesci, Paula Buttery
Increasing the number of parameters in language models is a common strategy to enhance their performance. However, smaller language models remain valuable due to their lower operat…
Prompting open-source and commercial language models for grammatical error correction of English learner text
Christopher Davis, Andrew Caines, Øistein Andersen +6
Thanks to recent advances in generative AI, we are able to prompt large language models (LLMs) to produce texts which are fluent and grammatical. In addition, it has been shown tha…