8 papers
Diagnosing Visual Ignorance in Vision-Language Models
Runyu Zhou, Qi Zhang, Qixun Wang +1
Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence. While this behavior is widely observed, i…
SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
Xiaojun Guo, Runyu Zhou, Yifei Wang +8
Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequatel…
On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy
Jingyi Cui, Qi Zhang, Yifei Wang +1
Sparse autoencoders (SAEs) have recently emerged as a powerful tool for interpreting the features learned by large language models (LLMs). By reconstructing features with sparsely…
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?
Zimo Wen, Boxiu Li, Wanbo Zhang +11
Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear. Existing benchmarks lac…
SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training
Qi Zhang, Yifei Wang, Xiaohan Wang +4
In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre-training, their effectiven…
An Augmentation Overlap Theory of Contrastive Learning
Qi Zhang, Yifei Wang, Yisen Wang
Recently, self-supervised contrastive learning has achieved great success on various tasks. However, its underlying working mechanism is yet unclear. In this paper, we first provid…