Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits

Lena Strobl

Transformers have emerged as a widely used neural network model for various natural language processing tasks. Previous research explored their relationship with constant-depth thr…