Attention Temperature Matters in Abstractive Summarization Distillation

Shengqiang Zhang, Xingxing Zhang, Hangbo Bao +1

Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims…