The Diminishing Returns of Masked Language Models to Science

Zhi Hong, Aswathy Ajith, Gregory Pauloski +3

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the dow…