ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention

Yang Liu, Jiaxiang Liu, Li Chen +7

Sparse Transformer has recently attracted a lot of attention since the ability for reducing the quadratic dependency on the sequence length. We argue that two factors, information…