8 papers
Priors in Time: Missing Inductive Biases for Language Model Interpretability
Ekdeep Singh Lubana, Can Rager, Sai Sumedh R. Hindupur +13
Recovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are ind…
Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting
Taha Binhuraib, Greta Tuckute, Nicholas Blauch
Spatial functional organization is a hallmark of biological brains: neurons are arranged topographically according to their response properties, at multiple scales. In contrast, re…
Modeling the language cortex with form-independent and enriched representations of sentence meaning reveals remarkable semantic abstractness
Shreya Saha, Shurui Li, Greta Tuckute +5
The human language system represents both linguistic forms and meanings, but the abstractness of the meaning representations remains debated. Here, we searched for abstract represe…
Representing Speech Through Autoregressive Prediction of Cochlear Tokens
Greta Tuckute, Klemen Kotar, Evelina Fedorenko +1
We introduce AuriStream, a biologically inspired model for encoding speech via a two-stage framework inspired by the human auditory processing hierarchy. The first stage transforms…
Language models align with brain regions that represent concepts across modalities
Maria Ryskina, Greta Tuckute, Alexander Fung +2
Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises…
Model Connectomes: A Generational Approach to Data-Efficient Language Models
Klemen Kotar, Greta Tuckute
Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks und…