5 papers · 1 filter
Do Syntax Heads Influence Model Behavior?
William Timkey, Tal Linzen
Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role i…
Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis
William Timkey, Brian Dillon, Tal Linzen
Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context, offering a potential link between human sentence processing and nex…
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
William Timkey, Tal Linzen
Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognit…
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
William Timkey, Marten van Schijndel
Similarity measures are a vital tool for understanding how language models represent and process language. Standard representational similarity measures such as cosine similarity a…
To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text
Matt Wilber, William Timkey, Marten Van Schijndel
Abstractive neural summarization models have seen great improvements in recent years, as shown by ROUGE scores of the generated summaries. But despite these improved metrics, there…