22 papers
From Markov to Laplace: How Mamba In-Context Learns Markov Chains
Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5
While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structur…
Diffuse AI Control on Fuzzy Tasks
Mikhail Terekhov, Caglar Gulcehre, Vivek Hebbar +1
AI models deployed in critical domains, such as AI safety research, may subtly sabotage our efforts due to misalignment. Diffuse AI Control is a subfield of AI safety concerned wit…
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
Justin Deschenaux, Caglar Gulcehre
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector…
Language Modeling with Hyperspherical Flows
Justin Deschenaux, Caglar Gulcehre
Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability,…
RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
Xiuying Wei, Caglar Gulcehre
Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of attention and the KV cache size by a factor of the dilation size D, while pres…
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
Xiuying Wei, Caglar Gulcehre
Efficient inference is critical for long-context language models, where attention computation and KV-cache access dominate the cost. Recent work RAT+, introduces a recurrence-augme…