5 papers
A Compositional Theory of Causally Masked Transformers
Franz Nowak, Ryan Cotterell, Reda Boumasmoud
What types of decision problems can a causally masked, finite-precision transformer solve for inputs of arbitrary length? Existing answers often rely on idealized arithmetic, but u…
Characterizing the Expressivity of Local Attention in Transformers
Jiaoda Li, Ryan Cotterell
The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate…
An Algebraic View of the Expressivity of Recurrent Language Models
Franz Nowak, Ryan Cotterell, Reda Boumasmoud
What formal languages can a recurrent neural language model recognize? Formal results in the literature conflict: some authors report Turing-completeness, while others show equival…
Bearing Syntactic Fruit with Stack-Augmented Neural Networks
Brian DuSell, Ryan Cotterell
When children learn language, they make syntactic generalizations based on hierarchical rules. A recent line of work has inquired as to whether common neural network architectures…
Transformers are Inherently Succinct
Pascal Bergsträßer, Ryan Cotterell, Anthony W. Lin
We study succinctness as a measure of the expressive power of transformers. Succinctness -- how compactly a formalism can describe a language relative to other formalisms -- is a c…