73 citations · 194 across the 74 of their papers we have counts for
87 papers
Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
Nurbüke Teker, Rui Xiao, Zeynep Akata +1
When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to res…
Task Vector Descent: Learning from Non-IID Batches
Anton Baumann, Jonas Hübotter, Zeynep Akata +1
A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when…
NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models
Omar Elfatairy, Maria A. Bravo, Jessica Bader +1
Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complementary ability to satisfy negat…
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
Luis Wiedmann, Leander Girrbach, Cordelia Schmid +1
Agent evaluations increasingly go beyond a single success rate, reporting metrics such as cost, consistency, and robustness. Yet they typically treat the agent itself as fixed. In…
Learning Functional Subspaces for Neural Network Compression
Massimo Bini, Anders Christensen, Stephan Alaniz +5
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus…
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3
Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this…
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Lukas Thede, Yash Kumar Atri, David Chen +4
Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose…
User Model Extraction via Belief Self-Distillation
Ali Holmov, Yiran Huang, Kirill Bykov +1
Large language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate.…
VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs
Jingyi He, Sanghwan Kim, Zeynep Akata
Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into…
From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA
Sena Korkut, María Alejandra Bravo Sarmiento, Sanghwan Kim +1
High benchmark accuracy does not guarantee genuine use of visual evidence. We study this problem in traffic accident Video Question Answering (VideoQA), where correct answers shoul…
APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings
Philipp Spohn, Leander Girrbach, Zeynep Akata
Typical LLM responses tend to follow a default style, even though users often have distinct preferences regarding tone, verbosity, and formality that they do not explicitly state i…
Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas
Nils A. Herrmann, Leander Girrbach, Kirill Bykov +1
Recent work shows that large language models (LLMs) encode behavioral traits ("personas") as linear directions in activation space, often called "persona vectors". Prior work has u…
Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
Yiran Huang, Lukas Thede, Massimiliano Mancini +2
While Large Vision Language Models (LVLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose deployment challenges on resource-const…
UNBOX: Unveiling Black-box visual models with Natural-language
Simone Carnemolla, Chiara Russo, Simone Palazzo +5
Ensuring trustworthiness in open-world visual recognition requires models that are interpretable, fair, and robust to distribution shifts. Yet modern vision systems are increasingl…
Explaining CLIP Zero-shot Predictions Through Concepts
Onat Ozdemir, Anders Christensen, Stephan Alaniz +2
Large-scale vision-language models such as CLIP have achieved remarkable success in zero-shot image recognition, yet their predictions remain largely opaque to human understanding.…
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Rui Xiao, Sanghwan Kim, Yongqin Xian +2
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coa…
From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs
Boyong Wu, Sanghwan Kim, Zeynep Akata
Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood. We…
Rethinking Concept Bottleneck Models: From Pitfalls to Solutions
Merve Tapli, Quentin Bouniot, Wolfgang Stammer +2
Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts but face fundamental limitations: the absence of a metric to pre-evaluate concept relevance, th…
The Latent Color Subspace: Emergent Order in High-Dimensional Chaos
Mateusz Pach, Jessica Bader, Quentin Bouniot +2
Text-to-image generation models have advanced rapidly, yet achieving fine-grained control over generated images remains difficult, largely due to limited understanding of how seman…
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Lukas Thede, Stefan Winzeck, Zeynep Akata +1
Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known…
Are Reasoning LLMs Robust to Interventions on Their Chain-of-Thought?
Alexander von Recum, Leander Girrbach, Zeynep Akata
Reasoning LLMs (RLLMs) generate step-by-step chains of thought (CoTs) before giving an answer, which improves performance on complex tasks and makes reasoning more transparent. But…
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
Simon Roschmann, Paul Krzakala, Sonia Mazelet +2
The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits thi…
Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps
Peter Holderrieth, Douglas Chen, Luca Eyring +7
Flow and diffusion models produce high-quality samples, but adapting them to user preferences or constraints post-training remains costly and brittle, a challenge commonly called r…
Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
Yiran Huang, Karsten Roth, Quentin Bouniot +2
Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate i…