Enabling Autoregressive Models to Fill In Masked Tokens

Daniel Israel, Aditya Grover, Guy Van den Broeck

Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR mo…