How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT${}^{\prime }$s Attention

Yue Guan, Jingwen Leng, Chao Li +2

Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mec…