9 papers
What is the Best Sequence Length for BABYLM?
Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely +1
Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, m…
BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models
Yuan Gao, Suchir Salhan, Andrew Caines +2
To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic S…
Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
Bianca-Mihaela Ganescu, Suchir Salhan, Andrew Caines +1
Training vision-language models on cognitively-plausible amounts of data requires rethinking how models integrate multimodal information. Within the constraints of the Vision track…
Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
David Demitri Africa, Suchir Salhan, Yuval Weiss +2
Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-cons…
Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
Yuval Weiss, David Demitri Africa, Paula Buttery +1
Parameter-efficient methods like LoRA have revolutionised large language model (LLM) fine-tuning. ReLoRA extends this idea to pretraining by repeatedly merging and reinitialising l…
BabyLM's First Words: Word Segmentation as a Phonological Probing Task
Zébulon Goriely, Paula Buttery
Language models provide a key framework for studying linguistic theories based on prediction, but phonological analysis using large language models (LLMs) is difficult; there are f…