1 citations · 1 across the 2 of their papers we have counts for
6 papers
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
James A. Michaelov, Roger P. Levy, Benjamin K. Bergen
We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregr…
Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett, Tyler A. Chang, Stella Biderman +1
The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less…
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
James A. Michaelov, Reeka Estacio, Zhien Zhang +1
Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show…
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
Tyler A. Chang, Benjamin K. Bergen
In Transformer language models, activation vectors transform from current token embeddings to next token predictions as they pass through the model. To isolate a minimal form of th…
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
Catherine Arnett, Tyler A. Chang, James A. Michaelov +1
Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood. We ask what happens to a monolingual language…
Why do language models perform worse for morphologically complex languages?
Catherine Arnett, Benjamin K. Bergen
Language models perform differently across languages. It has been previously suggested that morphological typology may explain some of this variability (Cotterell et al., 2018). We…