From the 1 of 5 linked papers with an AI index.
5 papers
AVQ-Attention: Adaptive Vector-Quantized Attention
Winfried van den dool, Patrick Forré, Amir Habibian +2
The paper introduces Adaptive Vector-Quantized (AVQ) Attention, which dynamically allocates codebook capacity to the most important regions of the key space, preserving O(MN) compl…
Knowledge Distillation for Visual Autoregressive Models
Elia Peruzzo, Aritra Bhowmik, Guillaume Sautiere +2
Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural ap…
Elastic ViTs from Pretrained Models without Retraining
Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort +2
Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world con…
Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning
Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort +1
Supervised fine-tuning (SFT) on chain-of-thought data is an essential post-training step for reasoning language models. Standard machine learning intuition suggests that training w…
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano
Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose…