11 papers
SVoT: State-aware Visualization-of-Thought for Spatial Reasoning via Reinforcement Learning
Chao Lei, Yanbei Jiang, Markus Hiller +4
Spatial reasoning remains a challenge for Multimodal Large Language Models (MLLMs), as it requires reliable multi-hop inference over both intermediate states and state transitions.…
Mind the Perspective: Let's Reason Recursively for Theory of Mind
Chao Lei, Guang Hu, Meng Yang +2
Theory of Mind (ToM) reasoning requires inferring agents' beliefs from partial and asymmetric observations, which remains an open challenge for LLMs. Existing prompting-based appro…
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
Yanbei Jiang, Amr Keleg, Ryandito Diandaru +4
While the real world is inherently stochastic, Large Language Models (LLMs) are predominantly evaluated on single-round inference against fixed ground truths. In this work, we shif…
Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning
Xueqi Ma, Shuo Yang, Yanbei Jiang +6
Despite remarkable advances in large Vision-Language Models (VLMs), spatial reasoning remains a persistent challenge. In this work, we investigate how attention heads within VLMs c…
Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules
Yanbei Jiang, Xueqi Ma, Shu Liu +5
Despite excelling on multimodal benchmarks, vision-language models (VLMs) largely remain a black box. In this paper, we propose a novel interpretability framework to systematically…
Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoning
Xueqi Ma, Jun Wang, Yanbei Jiang +3
Large language models (LLMs) have achieved state-of-the-art performance in a variety of tasks, but remain largely opaque in terms of their internal mechanisms. Understanding these…
PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning
Yanbei Jiang, Chao Lei, Yihao Ding +2
Despite significant progress, Vision-Language Models (VLMs) still struggle with complex visual reasoning, where multi-step dependencies cause early errors to cascade through the re…
Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysis
Xueqi Ma, Yanbei Jiang, Sarah Erfani +4
Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance across various objective multimodal perception tasks, yet their application to subjective, emotio…
Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
Yanbei Jiang, Yihao Ding, Chao Lei +3
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning…
Open-World Amodal Appearance Completion
Jiayang Ao, Yanbei Jiang, Qiuhong Ke +1
Understanding and reconstructing occluded objects is a challenging problem, especially in open-world scenarios where categories and contexts are diverse and unpredictable. Traditio…
KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph
Yanbei Jiang, Krista A. Ehinger, Jey Han Lau
Exploring the narratives conveyed by fine-art paintings is a challenge in image captioning, where the goal is to generate descriptions that not only precisely represent the visual…