SDBERT: SparseDistilBERT, a faster and smaller BERT model

Devaraju Vinoda, Pawan Kumar Yadav

In this work we introduce a new transformer architecture called SparseDistilBERT (SDBERT), which is a combination of sparse attention and knowledge distillantion (KD). We implement…