24 citations · 83 across the 12 of their papers we have counts for
10 papers · 1 filter
Feature Interactions Reveal Linguistic Structure in Language Models
Jaap Jumelet, Willem Zuidema
We study feature interactions in the context of feature attribution methods for post-hoc interpretability. In interpretability research, getting to grips with feature interactions…
Quantifying Context Mixing in Transformers
Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała +1
Self-attention weights and their transformed variants have been the main source of information for analyzing token-to-token interactions in Transformer-based models. But despite th…
The Birth of Bias: A case study on the evolution of gender bias in an English language model
Oskar van der Wal, Jaap Jumelet, Katrin Schulz +1
Detecting and mitigating harmful biases in modern language models are widely recognized as crucial, open problems. In this paper, we take a step back and investigate how language m…
Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations
Arabella Sinclair, Jaap Jumelet, Willem Zuidema +1
We investigate the extent to which modern, neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure…
Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment
Jaap Jumelet, Willem Zuidema, Dieuwke Hupkes
Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these re…
Formal models of Structure Building in Music, Language and Animal Songs
Willem Zuidema, Dieuwke Hupkes, Geraint Wiggins +2
Human language, music and a variety of animal vocalisations constitute ways of sonic communication that exhibit remarkable structural complexity. While the complexities of language…
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
Mario Giulianelli, Jacqueline Harding, Florian Mohnert +2
How do neural language models keep track of number agreement between subject and verb? We show that `diagnostic classifiers', trained to predict number from the internal states of…
Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity
Samira Abnar, Rasyan Ahmed, Max Mijnheer +1
We evaluate 8 different word embedding models on their usefulness for predicting the neural activation patterns associated with concrete nouns. The models we consider include an ex…
Unsupervised Dependency Parsing: Let's Use Supervised Parsers
Phong Le, Willem Zuidema
We present a self-training approach to unsupervised dependency parsing that reuses existing supervised and unsupervised parsing algorithms. Our approach, called `iterated reranking…
Compositional Distributional Semantics with Long Short Term Memory
Phong Le, Willem Zuidema
We are proposing an extension of the recursive neural network that makes use of a variant of the long short-term memory architecture. The extension allows information low in parse…