Average Attention Transformers and Arithmetic Circuits

Lena Ehrmuth, Laura Strieker

We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circ…