7 papers
Structural priors for data-efficient language learning
Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1
Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-lang…
Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
Jonas Mayer Martins, Ali Hamza Bashir, Muhammad Rehan Khalid +1
Children efficiently acquire language not just by listening, but by interacting with others in their social environment. Conversely, large language models are typically trained wit…
From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes
Zébulon Goriely, Richard Diehl Martinez, Andrew Caines +2
Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes ca…
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
Richard Diehl Martinez, Zebulon Goriely, Andrew Caines +2
Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize…
CLIMB: Curriculum Learning for Infant-inspired Model Building
Richard Diehl Martinez, Zebulon Goriely, Hope McGovern +4
We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge. The challenge requires training a language model from scratch using only a relatively small t…
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze
Ece Takmaz, Sandro Pezzelle, Lisa Beinborn +1
When speakers describe an image, they tend to look at objects before mentioning them. In this paper, we investigate such sequential cross-modal alignment by modelling the image des…
The Fluidity of Concept Representations in Human Brain Signals
Eva Hendrikx, Lisa Beinborn
Cognitive theories of human language processing often distinguish between concrete and abstract concepts. In this work, we analyze the discriminability of concrete and abstract con…