Masked ELMo: An evolution of ELMo towards fully contextual RNN language models

Gregory Senay, Emmanuelle Salin

This paper presents Masked ELMo, a new RNN-based model for language model pre-training, evolved from the ELMo language model. Contrary to ELMo which only uses independent left-to-r…