Of Non-Linearity and Commutativity in BERT

Sumu Zhao, Damian Pascual, Gino Brunner +1

In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-lin…