63 citations · 110 across the 16 of their papers we have counts for
3 papers · 1 filter
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity
Yiding Hao, Dana Angluin, Robert Frank
This paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard att…
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Aaron Mueller, Robert Frank, Tal Linzen +2
Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applicati…
Do Language Models Learn Position-Role Mappings?
Jackson Petty, Michael Wilson, Robert Frank
How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained…