5 citations · 9 across the 20 of their papers we have counts for
23 papers
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava +3
Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evid…
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Suchir Salhan, Catherine Arnett, James Michaelov +1
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure struc…
BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models
Shivam Singh, Aditya Yadavalli, Catherine Arnett +1
We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-b…
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks a…
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Ej Zhou, Suchir Salhan, Catherine Arnett +1
Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We a…
Skill Issue: Are Skills Language-Invariant in LLMs?
Bobby Cheng, Adam Gaber, Zhengyuan Liu +4
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work q…
Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
Stella Biderman, Mohammad Aflah Khan, Niloofar Mireshghallah +3
What would it mean to have a scientific understanding of AI? Models are not static objects: they are snapshots of time-evolving processes shaped by data, objectives, architectures,…
Weight Tying Biases Token Embeddings Towards the Output Space
Antonio Lopardo, Avyukth Harish, Catherine Arnett +1
Weight tying, i.e. sharing parameters between input and output embedding matrices, is common practice in language model design, yet its impact on the learned embedding space remain…
How Open Must Language Models be to Enable Reliable Scientific Inference?
James A. Michaelov, Catherine Arnett, Tyler A. Chang +7
How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions…
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…
Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
James A. Michaelov, Catherine Arnett
Language models generally produce grammatical text, but they are more likely to make errors in certain contexts. Drawing on paradigms from psycholinguistics, we carry out a fine-gr…
Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett, Tyler A. Chang, Stella Biderman +1
The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
Evaluating Morphological Alignment of Tokenizers in 70 Languages
Catherine Arnett, Marisa Hudspeth, Brendan O'Connor
While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One propose…
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood. We ask what happens to a monolingual language…
Why do language models perform worse for morphologically complex languages?
Catherine Arnett, Benjamin K. Bergen
Language models perform differently across languages. It has been previously suggested that morphological typology may explain some of this variability (Cotterell et al., 2018). We…
Goldfish: Monolingual Language Models for 350 Languages
Tyler A. Chang, Catherine Arnett, Zhuowen Tu +1
For many low-resource languages, the only available language models are large multilingual models trained on many languages simultaneously. Despite state-of-the-art performance on…
Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictabil…
Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement
Catherine Arnett, Pamela D. Rivière, Tyler A. Chang +1
The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in…
A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
Catherine Arnett, Tyler A. Chang, Benjamin K. Bergen
How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for…
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages
Tyler A. Chang, Catherine Arnett, Zhuowen Tu +1
Multilingual language models are widely used to extend NLP systems to low-resource languages. However, concrete evidence for the effects of multilinguality on language modeling per…
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
James A. Michaelov, Catherine Arnett, Tyler A. Chang +1
Abstract grammatical knowledge - of parts of speech and grammatical patterns - is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowl…
Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Do multilingual language models share abstract grammatical representations across languages, and if so, when do these develop? Following Sinclair et al. (2022), we use structural p…