Unsupervised Improvement of Factual Knowledge in Language Models

Nafis Sadeq, Byungkyu Kang, Prarit Lamba +1

Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for lear…