4 papers
Rare Text Semantics Were Always There in Your Diffusion Transformer
Seil Kang, Woojung Han, Dayun Ju +1
Starting from flow- and diffusion-based transformers, Multi-modal Diffusion Transformers (MM-DiTs) have reshaped text-to-vision generation, gaining acclaim for exceptional visual f…
Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
Jinyeong Kim, Seil Kang, Jiwoo Park +2
Large Vision-Language Models (LVLMs) answer visual questions by transferring information from images to text through a series of attention heads. While this image-to-text informati…
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
Seil Kang, Jinyeong Kim, Junhyeok Kim +1
Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVL…
See What You Are Told: Visual Attention Sink in Large Multimodal Models
Seil Kang, Jinyeong Kim, Junhyeok Kim +1
Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder. Ideally, these models should focus on k…