7 papers · 1 filter
Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMs
Moongyu Jeon, Dongjae Jeon, Bumjun Kim +2
Masked diffusion language models support arbitrary-order generation, suggesting a natural way to produce diverse outputs. However, recent work argues that this flexibility reduces…
DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs
Bumjun Kim, Dongjae Jeon, Moongyu Jeon +1
Parallel decoding for Diffusion LLMs (dLLMs) is difficult because each denoising step provides only token-wise marginal distributions, while unmasking multiple tokens simultaneousl…
Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs
Yoonjun Cho, Dongjae Jeon, Soeun Kim +2
Quantization Error Reconstruction (QER) reduces accuracy loss in Post-Training Quantization (PTQ) by approximating weights as …
Information-Theoretic Discrete Diffusion
Moongyu Jeon, Sangwoo Shin, Dongjae Jeon +1
We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE…
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Yoonjun Cho, Soeun Kim, Dongjae Jeon +3
Decomposing weight matrices into quantization and low-rank components () is a widely used technique for compressing large lang…
Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes
Dongjae Jeon, Dueun Kim, Albert No
In this paper, we introduce a geometric framework to analyze memorization in diffusion models through the sharpness of the log probability density. We mathematically justify a prev…
An Information Theoretic Evaluation Metric For Strong Unlearning
Dongjae Jeon, Wonje Jeung, Taeheon Kim +2
Machine unlearning (MU) aims to remove the influence of specific data from trained models, addressing privacy concerns and ensuring compliance with regulations such as the ``right…