Telling BERT's full story: from Local Attention to Global Aggregation

Damian Pascual, Gino Brunner, Roger Wattenhofer

We take a deep look into the behavior of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explainin…