4 papers
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1
Post-training hybridization of large language models (LLMs) often replaces quadratic self-attention with sliding-window attention (SWA) to reduce KV cache usage and improve latency…
Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs
Linrui Ma, Yufei Cui, Kai Han +1
Discrete diffusion models offer global context awareness and flexible parallel generation. However, uniform random noise schedulers in standard DLLM training overlook the highly no…
Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
Linrui Ma, Yufei Cui, Kai Han +1
One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studie…
Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants
Yunhe Wang, Kai Han, Huiling Zhen +13
The paradigm of Large Language Models (LLMs) is currently defined by auto-regressive (AR) architectures, which generate text through a sequential ``brick-by-brick'' process. Despit…