Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models

Muhammed Shahir Abdurrahman, Hashem Elezabi, Bruce Changlong Xu

The choice of \emph{which} tokens to mask is a central, under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, but…