5 citations · 9 across the 11 of their papers we have counts for
13 papers
How Open Must Language Models be to Enable Reliable Scientific Inference?
James A. Michaelov, Catherine Arnett, Tyler A. Chang +7
How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions…
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
James A. Michaelov, Roger P. Levy, Benjamin K. Bergen
We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregr…
Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett, Tyler A. Chang, Stella Biderman +1
The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less…
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
James A. Michaelov, Reeka Estacio, Zhien Zhang +1
Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show…
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
Tyler A. Chang, Benjamin K. Bergen
In Transformer language models, activation vectors transform from current token embeddings to next token predictions as they pass through the model. To isolate a minimal form of th…
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood. We ask what happens to a monolingual language…
Why do language models perform worse for morphologically complex languages?
Catherine Arnett, Benjamin K. Bergen
Language models perform differently across languages. It has been previously suggested that morphological typology may explain some of this variability (Cotterell et al., 2018). We…
Goldfish: Monolingual Language Models for 350 Languages
Tyler A. Chang, Catherine Arnett, Zhuowen Tu +1
For many low-resource languages, the only available language models are large multilingual models trained on many languages simultaneously. Despite state-of-the-art performance on…
Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictabil…
A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
Catherine Arnett, Tyler A. Chang, Benjamin K. Bergen
How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for…
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages
Tyler A. Chang, Catherine Arnett, Zhuowen Tu +1
Multilingual language models are widely used to extend NLP systems to low-resource languages. However, concrete evidence for the effects of multilinguality on language modeling per…
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
James A. Michaelov, Catherine Arnett, Tyler A. Chang +1
Abstract grammatical knowledge - of parts of speech and grammatical patterns - is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowl…
Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Do multilingual language models share abstract grammatical representations across languages, and if so, when do these develop? Following Sinclair et al. (2022), we use structural p…