163 citations · 1.4k across the 139 of their papers we have counts for
8 papers · 2 filters
IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning
Soumya Suvra Ghosal, Samyadeep Basu, Soheil Feizi +1
Image-text contrastive models such as CLIP learn transferable and robust representations for zero-shot transfer to a variety of downstream tasks. However, to obtain strong downstre…
Understanding Information Storage and Transfer in Multi-modal Large Language Models
Samyadeep Basu, Martin Grayson, Cecily Morrison +3
Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these me…
Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings
Arman Zarei, Keivan Rezaei, Samyadeep Basu +4
Text-to-image diffusion-based generative models have the stunning ability to generate photo-realistic images and achieve state-of-the-art low FID scores on challenging image genera…
Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
Sriram Balasubramanian, Samyadeep Basu, Soheil Feizi
Recent work has explored how individual components of the CLIP-ViT model contribute to the final representation by leveraging the shared image-text representation space of CLIP. Th…
On Mechanistic Knowledge Localization in Text-to-Image Generative Models
Samyadeep Basu, Keivan Rezaei, Priyatham Kattakinda +5
Identifying layers within text-to-image models which control visual attributes can facilitate efficient model editing through closed-form updates. Recent work, leveraging causal tr…
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning
Neha Kalibhat, Priyatham Kattakinda, Sumit Nawathe +5
Vision transformers have established a precedent of patchifying images into uniformly-sized chunks before processing. We hypothesize that this design choice may limit models in lea…
Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models
Mazda Moayeri, Samyadeep Basu, Sriram Balasubramanian +4
Recent text-to-image generative models such as Stable Diffusion are extremely adept at mimicking and generating copyrighted content, raising concerns amongst artists that their uni…
What do we learn from inverting CLIP models?
Hamid Kazemi, Atoosa Chegini, Jonas Geiping +2
We employ an inversion-based approach to examine CLIP models. Our examination reveals that inverting CLIP models results in the generation of images that exhibit semantic alignment…