Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

Muhammed Shahir Abdurrahman, Hashem Elezabi, Bruce Changlong Xu

The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aw…