5 papers
AVQ-Attention: Adaptive Vector-Quantized Attention
Winfried van den dool, Patrick Forré, Amir Habibian +2
The complexity of attention over tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O…
Knowledge Distillation for Visual Autoregressive Models
Elia Peruzzo, Aritra Bhowmik, Guillaume Sautiere +2
Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural ap…
Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning
Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort +1
Supervised fine-tuning (SFT) on chain-of-thought data is an essential post-training step for reasoning language models. Standard machine learning intuition suggests that training w…
Elastic ViTs from Pretrained Models without Retraining
Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort +2
Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world con…
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano
Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose…