76 citations · 111 across the 20 of their papers we have counts for
20 papers
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Suchir Salhan, Catherine Arnett, James Michaelov +1
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure struc…
How Open Must Language Models be to Enable Reliable Scientific Inference?
James A. Michaelov, Catherine Arnett, Tyler A. Chang +7
How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions…
N-gram-like Language Models Predict Naturalistic Reading Time Best
James A. Michaelov, Roger P. Levy
Recent work has found that contemporary language models such as transformers can become so good at next-word prediction that the probabilities they calculate become worse for predi…
Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs
Sean Trott, Samuel Taylor, Cameron Jones +2
Research on mental state reasoning in language models (LMs) has the potential to inform theories of human social cognition--such as the theory that mental state reasoning emerges i…
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
James A. Michaelov, Roger P. Levy, Benjamin K. Bergen
We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregr…
Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
James A. Michaelov, Catherine Arnett
Language models generally produce grammatical text, but they are more likely to make errors in certain contexts. Drawing on paradigms from psycholinguistics, we carry out a fine-gr…
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
James A. Michaelov, Reeka Estacio, Zhien Zhang +1
Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show…
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood. We ask what happens to a monolingual language…
Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictabil…
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
James A. Michaelov, Catherine Arnett, Tyler A. Chang +1
Abstract grammatical knowledge - of parts of speech and grammatical patterns - is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowl…
Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Do multilingual language models share abstract grammatical representations across languages, and if so, when do these develop? Following Sinclair et al. (2022), we use structural p…
Emergent inabilities? Inverse scaling over the course of pretraining
James A. Michaelov, Benjamin K. Bergen
Does inverse scaling only occur as a function of model size, or can it also occur over the course of training? We carry out an exploratory study investigating whether the performan…
Can Peanuts Fall in Love with Distributional Semantics?
James A. Michaelov, Seana Coulson, Benjamin K. Bergen
Context changes expectations about upcoming words - following a story involving an anthropomorphic peanut, comprehenders expect the sentence the peanut was in love more than the pe…
Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers
James A. Michaelov, Benjamin K. Bergen
How well do language models deal with quantification? In this study, we focus on 'few'-type quantifiers, as in 'few children like toys', which might pose a particular challenge for…
Collateral facilitation in humans and language models
James A. Michaelov, Benjamin K. Bergen
Are the predictions of humans and language models affected by similar things? Research suggests that while comprehending language, humans make predictions about upcoming words, wit…
Do Large Language Models know what humans know?
Sean Trott, Cameron Jones, Tyler Chang +2
Humans can attribute beliefs to others. However, it is unknown to what extent this ability results from an innate biological endowment or from experience accrued through child deve…
Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns?
James A. Michaelov, Benjamin K. Bergen
Some languages allow arguments to be omitted in certain contexts. Yet human language comprehenders reliably infer the intended referents of these zero pronouns, in part because the…
So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements
James A. Michaelov, Seana Coulson, Benjamin K. Bergen
More predictable words are easier to process - they are read faster and elicit smaller neural signals associated with processing difficulty, most notably, the N400 component of the…
Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?
James A. Michaelov, Megan D. Bardolph, Seana Coulson +1
Despite being designed for performance rather than cognitive plausibility, transformer language models have been found to be better at predicting metrics used to assess human langu…
How well does surprisal explain N400 amplitude under different experimental conditions?
James A. Michaelov, Benjamin K. Bergen
We investigate the extent to which word surprisal can be used to predict a neural measure of human language processing difficulty - the N400. To do this, we use recurrent neural ne…