5 citations · 9 across the 20 of their papers we have counts for
10 papers · 1 filter
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava +3
Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evid…
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Suchir Salhan, Catherine Arnett, James Michaelov +1
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure struc…
BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models
Shivam Singh, Aditya Yadavalli, Catherine Arnett +1
We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-b…
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks a…
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Ej Zhou, Suchir Salhan, Catherine Arnett +1
Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We a…
Skill Issue: Are Skills Language-Invariant in LLMs?
Bobby Cheng, Adam Gaber, Zhengyuan Liu +4
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work q…
Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
Stella Biderman, Mohammad Aflah Khan, Niloofar Mireshghallah +3
What would it mean to have a scientific understanding of AI? Models are not static objects: they are snapshots of time-evolving processes shaped by data, objectives, architectures,…
Weight Tying Biases Token Embeddings Towards the Output Space
Antonio Lopardo, Avyukth Harish, Catherine Arnett +1
Weight tying, i.e. sharing parameters between input and output embedding matrices, is common practice in language model design, yet its impact on the learned embedding space remain…
How Open Must Language Models be to Enable Reliable Scientific Inference?
James A. Michaelov, Catherine Arnett, Tyler A. Chang +7
How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions…
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…