Simulating Hard Attention Using Soft Attention

Andy Yang, Lena Strobl, David Chiang +1

We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine…