UT5: Pretraining Non autoregressive T5 with unrolled denoising

Mahmoud G. Salem, Jiayu Ye, Chu-Cheng Lin +1

Recent advances in Transformer-based Large Language Models have made great strides in natural language generation. However, to decode K tokens, an autoregressive model needs K sequ…