1 paper
Tomas Ruiz, Zhen Qin, Yifan Zhang +3
Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We…