18 papers
Weak type estimates for Riesz transforms of elliptic operators and magnetic Schrödinger operators
El Maati Ouhabaz
Let be a complete Riemannian manifold and define a magnetic Schrödinger operator $\LL = ({\bf d}+ iA)^*({\bf d}+iA) + V$ where is a real -differential form in $L^2_{loc}…
Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack
Liwei Che, Yihao Quan, Sen Fang +4
Multimodal Large Language Models (MLLMs) consistently struggle with fine-grained visual counting, yet the underlying causes remain poorly understood. In this work, we present a mec…
Personalize Your Large Vision-language Models With In-context Prompt Tuning
Yanshu Li, Jiaqian Li, Kuai Yu +4
Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing inter…
Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models
Liwei Che, Zhiyu Xue, Yihao Quan +7
Counting serves as a simple but powerful test of a Large Vision-Language Model's (LVLM's) reasoning; it forces the model to identify each individual object and then add them all up…
Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model
Yanchuan Tang, Taowen Wang, Yuefei Chen +3
Vision-Language-Action (VLA) models enable general-purpose robotic policies by mapping visual observations and language instructions to low-level actions, but they often lack relia…
Improving Visual Reasoning with Iterative Evidence Refinement
Zeru Shi, Kai Mei, Yihao Quan +2
Vision language models (VLMs) are increasingly capable of reasoning over images, but robust visual reasoning often requires re-grounding intermediate steps in the underlying visual…
Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding
Yuefei Chen, Jiang Liu, Xiaodong Lin +1
Vision Language Models (VLMs) have recently shown significant advancements in video understanding, especially in feature alignment, event reasoning, and instruction-following tasks…
Read the Scene, Not the Script: Outcome-Aware Safety for LLMs
Rui Wu, Yihao Quan, Zeru Shi +3
Safety-aligned Large Language Models (LLMs) still show two dominant failure modes: they are easily jailbroken, or they over-refuse harmless inputs that contain sensitive surface si…
Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning
Zeru Shi, Yingjia Wan, Zhenting Wang +4
Motivated by the puzzling observation that inserting long sequences of meaningless tokens before the query prompt can consistently enhance LLM reasoning performance, this work anal…
Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention
Jiaqian Li, Yanshu Li, Ligong Han +2
Implicit in-context learning (ICL) has newly emerged as a promising paradigm that simulates ICL behaviors in the representation space of large language models (LLMs), aiming to att…
CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning
Yanshu Li, Jianjiang Yang, Zhennan Shen +3
Modern large vision-language models (LVLMs) convert each input image into a large set of tokens that far outnumber the text tokens. Although this improves visual perception, it als…
TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration
Yanshu Li, Jianjiang Yang, Tian Yun +3
Multimodal in-context learning (ICL) has emerged as a key mechanism for harnessing the capabilities of large vision-language models (LVLMs). However, its effectiveness remains high…
Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning
Yanshu Li, Jianjiang Yang, Ziteng Yang +8
Multimodal in-context learning (ICL) is becoming a key capability that allows large vision-language models (LVLMs) to adapt to novel tasks without parameter updates, which expands…
MIV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
Yanshu Li, Yi Cao, Hongyang He +5
Multimodal in-context learning (ICL) equips Large Vision-language Models (LVLMs) with the ability to adapt to new tasks via multiple user-provided demonstrations, without requiring…
Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs
Liwei Che, Tony Qingze Liu, Jing Jia +3
Despite their remarkable potential, Large Vision-Language Models (LVLMs) still face challenges with object hallucination, a problem where their generated outputs mistakenly incorpo…
Can Large Vision-Language Models Detect Images Copyright Infringement from GenAI?
Qipan Xu, Zhenting Wang, Xiaoxiao He +2
Generative AI models, renowned for their ability to synthesize high-quality content, have sparked growing concerns over the improper generation of copyright-protected material. Whi…
Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
Yanbo Fang, Ruixiang Tang
Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes…
Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models
Yuqing Zhou, Ruixiang Tang, Ziyu Yao +1
Language models (LMs), despite their advances, often depend on spurious correlations, undermining their accuracy and generalizability. This study addresses the overlooked impact of…