7 papers
Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Arnas Uselis, Andrea Dittadi, Seong Joon Oh
Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massiv…
When Do Diffusion Models learn to Generate Multiple Objects?
Yujin Jeong, Arnas Uselis, Iro Laina +2
Text-to-image diffusion models achieve impressive visual fidelity, yet they remain unreliable in multi-object generation. Despite extensive empirical evidence of these failures, th…
How can embedding models bind concepts?
Arnas Uselis, Darina Koishigarina, Seong Joon Oh
Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with…
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1
Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream…
Half-Truths Break Similarity-Based Retrieval
Bora Kargi, Arnas Uselis, Seong Joon Oh
When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this i…
Enhancing Multi-Image Understanding through Delimiter Token Scaling
Minyoung Lee, Yeji Park, Dongjun Hwang +3
Large Vision-Language Models (LVLMs) achieve strong performance on single-image tasks, but their performance declines when multiple images are provided as input. One major reason i…