Contextual Representation Learning beyond Masked Language Modeling

Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu +2

How do masked language models (MLMs) such as BERT learn contextual representations? In this work, we analyze the learning dynamics of MLMs. We find that MLMs adopt sampled embeddin…