7 papers
Fractional State Space Transition for Long Sequence Modeling
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Mos…
Survival-Guided Length Control for Efficient Diffusion Language Models
Ivan Kobyzev, Abbas Ghaddar, Yufei Cui
Diffusion language models (DLMs) generate text by iteratively denoising masked sequences, but standard decoding either fixes the sequence length or relies on ad hoc stopping rules,…
Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models
Saba Ahmadi, Prasanna Parthasarathi, Yufei Cui
Diffusion language models are a promising alternative to autoregressive models, yet post-training methods for them largely adapt reward-maximizing objectives. We identify a central…
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1
Post-training hybridization of large language models (LLMs) often replaces quadratic self-attention with sliding-window attention (SWA) to reduce KV cache usage and improve latency…
Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs
Linrui Ma, Yufei Cui, Kai Han +1
Discrete diffusion models offer global context awareness and flexible parallel generation. However, uniform random noise schedulers in standard DLLM training overlook the highly no…
Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
Linrui Ma, Yufei Cui, Kai Han +1
One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studie…
Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants
Yunhe Wang, Kai Han, Huiling Zhen +13
The paradigm of Large Language Models (LLMs) is currently defined by auto-regressive (AR) architectures, which generate text through a sequential ``brick-by-brick'' process. Despit…