Mixing Times of Glauber Dynamics on Masked Language Models

Suvadip Sana, Sami Wolf, Neer Mehta +4

Masked language models (MLMs) define local conditional distributions over tokens but do not, in general, correspond to any consistent joint distribution over sequences. This raises…