12 papers
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Suchir Salhan, Catherine Arnett, James Michaelov +1
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure struc…
LangMAP: A Language-Adaptive Approach to Tokenization
Clara Meister, Suchir Salhan, Andrzej Szablewski +3
Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing pretrained model requires either t…
LLMs Contain Multitudes: How Deployment Context Reshapes Model-Level Preferences and Values
Filip Trhlik, Aoife O'Flynn, Angela Yu +2
Large language models (LLMs) are increasingly characterised in recent evaluation work as having stable, model-level preference and value systems. However, accompanying robustness c…
What is the Best Sequence Length for BABYLM?
Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely +1
Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, m…
BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models
Yuan Gao, Suchir Salhan, Andrew Caines +2
To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic S…
Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
David Demitri Africa, Suchir Salhan, Yuval Weiss +2
Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-cons…
Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
Yuval Weiss, David Demitri Africa, Paula Buttery +1
Parameter-efficient methods like LoRA have revolutionised large language model (LLM) fine-tuning. ReLoRA extends this idea to pretraining by repeatedly merging and reinitialising l…
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
Richard Diehl Martinez, David Demitri Africa, Yuval Weiss +3
Building language models (LMs), especially small and medium ones, remains more art than science. While large LMs often improve by sheer scale, it is still unclear why many design c…
Learning Dynamics of Meta-Learning in Small Model Pretraining
David Demitri Africa, Yuval Weiss, Paula Buttery +1
Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integr…
ByteSpan: Information-Driven Subword Tokenisation
Zébulon Goriely, Suchir Salhan, Pietro Lesci +2
Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentati…
BabyLM's First Words: Word Segmentation as a Phonological Probing Task
Zébulon Goriely, Paula Buttery
Language models provide a key framework for studying linguistic theories based on prediction, but phonological analysis using large language models (LLMs) is difficult; there are f…
IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling
Zébulon Goriely, Paula Buttery
In this paper, we introduce two resources: (i) G2P+, a tool for converting orthographic datasets to a consistent phonemic representation; and (ii) IPA CHILDES, a phonemic dataset o…