Using Selective Masking as a Bridge between Pre-training and Fine-tuning

Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal +1

Pre-training a language model and then fine-tuning it for downstream tasks has demonstrated state-of-the-art results for various NLP tasks. Pre-training is usually independent of t…