22 citations · 25 across the 5 of their papers we have counts for
5 papers
Concise One-Layer Transformers Can Perform Dictionary Lookup
Lena Strobl, Anton Hampe, Dana Angluin +1
While transformers have proven enormously successful in a range of tasks, their fundamental properties as models of computation are not well understood. This paper contributes to t…
Simulating Hard Attention Using Soft Attention
Andy Yang, Lena Strobl, David Chiang +1
We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine…
Transformers as Transducers
Lena Strobl, Dana Angluin, David Chiang +2
We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transduction…
What Formal Languages Can Transformers Express? A Survey
Lena Strobl, William Merrill, Gail Weiss +2
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problem…
Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits
Lena Strobl
Transformers have emerged as a widely used neural network model for various natural language processing tasks. Previous research explored their relationship with constant-depth thr…