Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling

Ilia Kulikov, Maksim Eremeev, Kyunghyun Cho

Neural autoregressive sequence models smear the probability among many possible sequences including degenerate ones, such as empty or repetitive sequences. In this work, we tackle…