5 papers
Investigating Associational Biases in Inter-Model Communication of Large Generative Models
Fethiye Irmak Dogan, Yuval Weiss, Kajal Patel +2
Social bias in generative AI can manifest not only as performance disparities but also as associational bias, whereby models learn and reproduce stereotypical associations between…
Learning Dynamics of Meta-Learning in Small Model Pretraining
David Demitri Africa, Yuval Weiss, Paula Buttery +1
Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integr…
Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
David Demitri Africa, Suchir Salhan, Yuval Weiss +2
Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-cons…
Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
Yuval Weiss, David Demitri Africa, Paula Buttery +1
Parameter-efficient methods like LoRA have revolutionised large language model (LLM) fine-tuning. ReLoRA extends this idea to pretraining by repeatedly merging and reinitialising l…
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
Richard Diehl Martinez, David Demitri Africa, Yuval Weiss +3
Building language models (LMs), especially small and medium ones, remains more art than science. While large LMs often improve by sheer scale, it is still unclear why many design c…