565 citations · 818 across the 51 of their papers we have counts for
9 papers · 2 filters
Sparse Interventions in Language Models with Differentiable Masking
Nicola De Cao, Leon Schmid, Dieuwke Hupkes +1
There has been a lot of interest in understanding what information is captured by hidden representations of language models (LMs). Typically, interpretation methods i) do not guara…
How BPE Affects Memorization in Transformers
Eugene Kharitonov, Marco Baroni, Dieuwke Hupkes
Training data memorization in NLP can both be beneficial (e.g., closed-book QA) and undesirable (personal data extraction). In any case, successful model training requires a non-tr…
Towards Interactive Language Modeling
Maartje ter Hoeve, Evgeny Kharitonov, Dieuwke Hupkes +1
Interaction between caregivers and children plays a critical role in human language acquisition and development. Given this observation, it is remarkable that explicit interaction…
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
Yair Lakretz, Théo Desbordes, Dieuwke Hupkes +1
Recursive processing is considered a hallmark of human linguistic abilities. A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed…
The paradox of the compositionality of natural language: a neural machine translation case study
Verna Dankers, Elia Bruni, Dieuwke Hupkes
Obtaining human-like performance in NLP is often argued to require compositional generalisation. Whether neural networks exhibit this ability is usually studied by training models…
Language Models Use Monotonicity to Assess NPI Licensing
Jaap Jumelet, Milica Denić, Jakub Szymanik +2
We investigate the semantic knowledge of language models (LMs), focusing on (1) whether these LMs create categories of linguistic environments based on their semantic monotonicity…
Attention vs non-attention for a Shapley-based explanation method
Tom Kersten, Hugh Mee Wong, Jaap Jumelet +1
The field of explainable AI has recently seen an explosion in the number of explanation methods for highly non-linear deep neural networks. The extent to which such methods -- that…
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
Koustuv Sinha, Robin Jia, Dieuwke Hupkes +3
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent i…
Language Modelling as a Multi-Task Problem
Lucas Weber, Jaap Jumelet, Elia Bruni +1
In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability.…