13 papers
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representati…
Failing to See or Failing to Know? Attributing Errors in Vision-Language Models
Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2
Vision-language models (VLMs) can recognize entities in clear images yet still fail when answering questions that require factual knowledge beyond what is directly observable. Prio…
CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts
Shanu Kumar, Shubhanshu Khandelwal, Akhila Yesantarao Venkata +3
Prompts tuned for accuracy often grow long, raising inference cost on every model call. The best accuracy-cost trade-off depends on the task and the budget, so prompt optimization…
Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech
Adarsh Arigala, Arjun Gangwar, S Umesh +1
Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding. Grounding text in its visual fo…
LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar +2
Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetti…
VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Haz Sameen Shahgir, Xiaofu Chen, Yu Fu +4
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perce…
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation
Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva
As interest grows in generating long, detailed image captions, standard evaluation metrics become increasingly unreliable. N-gram-based metrics though efficient, fail to capture se…
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
Israfel Salazar, Desmond Elliott, Yova Kementchedjhieva
Contrastive vision-language models (VLMs) have made significant progress in binding visual and textual information, yet understanding long, compositional captions remains an open c…
The Devil is in the EOS: Sequence Training for Detailed Image Captioning
Abdelrahman Mohamed, Yova Kementchedjhieva
Despite significant advances in vision-language models (VLMs), image captioning often suffers from a lack of detail, with base models producing short, generic captions. This limita…
CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
George Ibrahim, Rita Ramos, Yova Kementchedjhieva
Multilingual vision-language models have made significant strides in image captioning, yet they still lag behind their English counterparts due to limited multilingual training dat…
LLMs Can Compensate for Deficiencies in Visual Representations
Sho Takishita, Jay Gala, Abdelrahman Mohamed +2
Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We inv…
Overcoming Vocabulary Constraints with Pixel-level Fallback
Jonas F. Lotz, Hendra Setiawan, Stephan Peitz +1
Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized durin…
JEEM: Vision-Language Understanding in Four Arabic Dialects
Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali +7
We introduce JEEM, a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Mo…