Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding

Shuohang Wang, Luowei Zhou, Zhe Gan +5

Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows fully-connected con…