Transformer Heads Looking for Order

Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser +4

In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transforme…