6 papers
Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions
Anastasiia Sedova, Natalie Schluter, Skyler Seto +1
Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is s…
Scaling Laws for Mixture Pretraining Under Data Constraints
Anastasiia Sedova, Skyler Seto, Natalie Schluter +1
As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in si…
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
Paul Jeha, Anastasiia Sedova, Louis Béthune +4
For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization…
How Value Induction Reshapes LLM Behaviour
Arnav Arora, Natalie Schluter, Katherine Metcalf +1
Conversational Large Language Models are post-trained on language that expresses specific behavioural traits, such as curiosity, open-mindedness, and empathy, and values, such as h…
Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks
Maureen de Seyssel, Jie Chi, Skyler Seto +3
We introduce a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). I…
The Role of Prosody in Spoken Question Answering
Jie Chi, Maureen de Seyssel, Natalie Schluter
Spoken language understanding research to date has generally carried a heavy text perspective. Most datasets are derived from text, which is then subsequently synthesized into spee…