Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction

Xiaojie Xia, Huigang Zhang, Chaoliang Zhong +2

Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practica…