Improving BERT with Self-Supervised Attention

Yiren Chen, Xiaoyu Kou, Jiangang Bai +1

One of the most popular paradigms of applying large pre-trained NLP models such as BERT is to fine-tune it on a smaller dataset. However, one challenge remains as the fine-tuned mo…