4 papers
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Minsik Choi, Geewook Kim, Young Geun Kim
Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output…
CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformers
Yonglak Son, Suhyeok Kim, Seungryong Kim +1
Diffusion transformer (DiT) achieves remarkable performance in visual generation, but its iterative denoising process combined with larger capacity leads to a high inference cost.…
First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training
Gyudong Kim, Hyukju Na, Jin Hyeon Kim +6
As training billion-scale transformers becomes increasingly common, employing multiple distributed GPUs along with parallel training methods has become a standard practice. However…
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Minsik Choi, Hyegang Son, Changhoon Kim +1
Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency chall…