Blockwise Self-Attention for Long Document Understanding

Jiezhong Qiu, Hao Ma, Omer Levy +3

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the…