Information-Theoretic Discrete Diffusion

Moongyu Jeon, Sangwoo Shin, Dongjae Jeon +1

We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE…