Contextual BERT: Conditioning the Language Model Using a Global State

Timo I. Denk, Ana Peleteiro Ramallo

BERT is a popular language model whose main pre-training task is to fill in the blank, i.e., predicting a word that was masked out of a sentence, based on the remaining words. In s…