Masked Language Model Scoring

Julian Salazar, Davis Liang, Toan Q. Nguyen +1

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are co…