6 papers
Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph +2
Training vision language models (VLMs) aims to align visual representations from a vision encoder with the textual representations of a pretrained large language model (LLM). Howev…
Articulate3D: Zero-Shot Text-Driven 3D Object Posing
Oishi Deb, Anjun Hu, Ashkan Khakzar +2
We propose a training-free method, Articulate3D, to pose a 3D asset through language control. Despite advances in vision and language models, this task remains surprisingly challen…
RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching
Farnoush Rezaei Jafari, Oliver Eberle, Ashkan Khakzar +1
Activation patching is a standard method in mechanistic interpretability for localizing the components of a model responsible for specific behaviors, but it is computationally expe…
Minimalist Concept Erasure in Generative Models
Yang Zhang, Er Jin, Yanfei Dong +5
Recent advances in generative models have demonstrated remarkable capabilities in producing high-quality images, but their reliance on large-scale unlabeled data has raised signifi…
How Visual Representations Map to Language Feature Space in Multimodal LLMs
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph +2
Effective multimodal reasoning depends on the alignment of visual and linguistic representations, yet the mechanisms by which vision-language models (VLMs) achieve this alignment r…
Mixture of Experts Made Intrinsically Interpretable
Xingyi Yang, Constantin Venhoff, Ashkan Khakzar +4
Neurons in large language models often exhibit \emph{polysemanticity}, simultaneously encoding multiple unrelated concepts and obscuring interpretability. Instead of relying on pos…