37 citations · 78 across the 32 of their papers we have counts for
36 papers
Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking
Haoyang Luo, Linwei Tao, Jie Gui +4
Multimodal Large Language Models (MLLMs) suffer from hallucinations, creating a critical need for Uncertainty Quantification (UQ) to ensure reliable deployment. However, existing a…
Region-Level Policy Optimization for Fine-grained MLLM Perception
Yuheng Shi, Xiaohuan Pei, Minjing Dong +1
Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual tokens inflate vision-encoding and language-model prefilling costs. We…
RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
Qiyanhui Lu, Han Wu, Rongjian Xu +6
Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive. Existing training-free pruning methods sele…
From Question Answering to Task Completion: A Survey on Agent System and Harness Design
Jianyuan Guo, Zhiwei Hao, Chengcheng Wang +14
LLM-based agents mark a shift from passive question answering to active task completion: they perceive environments, invoke tools, maintain state, and act over extended horizons. A…
Retrieval-Augmented Linguistic Calibration
Yi-Fan Yeh, Linwei Tao, Minjing Dong +4
Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic c…
Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
Yuheng Shi, Xiaohuan Pei, Linfeng Wen +2
MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indi…
Confidence Calibration under Ambiguous Ground Truth
Linwei Tao, Haoyang Luo, Minjing Dong +1
Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-vote…
PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
Hefei Mei, Zirui Wang, Chang Xu +2
Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box…
Can Large Language Models Express Uncertainty Like Human?
Linwei Tao, Yi-Fan Yeh, Bo Kai +6
Large language models (LLMs) are increasingly used in high-stakes settings, where overconfident responses can mislead users. Reliable confidence estimation has been shown to enhanc…
Efficient Rectified Flow for Image Fusion
Zirui Wang, Jiayi Zhang, Tianwei Guan +4
Image fusion is a fundamental and important task in computer vision, aiming to combine complementary information from different modalities to fuse images. In recent years, diffusio…
Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
Yuheng Shi, Xiaohuan Pei, Minjing Dong +1
Multimodal Large Language Models (MLLMs) require high-resolution visual information to perform fine-grained perception, yet processing entire high-resolution images is computationa…
WATS: Calibrating Graph Neural Networks with Wavelet-Aware Temperature Scaling
Xiaoyang Li, Linwei Tao, Haohui Lu +3
Graph Neural Networks (GNNs) have demonstrated strong predictive performance on relational data; however, their confidence estimates often misalign with actual predictive correctne…
Sample Margin-Aware Recalibration of Temperature Scaling
Haolan Guo, Linwei Tao, Haoyang Luo +2
Recent advances in deep learning have significantly improved predictive accuracy. However, modern neural networks remain systematically overconfident, posing risks for deployment i…
Revisiting Uncertainty Estimation and Calibration of Large Language Models
Linwei Tao, Yi-Fan Yeh, Minjing Dong +3
As large language models (LLMs) are increasingly deployed in high-stakes applications, robust uncertainty estimation is essential for ensuring the safe and trustworthy deployment o…
Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Haoyang Luo, Linwei Tao, Minjing Dong +1
Model calibration seeks to ensure that models produce confidence scores that accurately reflect the true likelihood of their predictions being correct. However, existing calibratio…
VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models
Hefei Mei, Zirui Wang, Shen You +2
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding and generation, yet their vulnerability to adversarial attacks raises sig…
Uncertainty Weighted Gradients for Model Calibration
Jinxu Lin, Linwei Tao, Minjing Dong +1
Model calibration is essential for ensuring that the predictions of deep neural networks accurately reflect true probabilities in real-world classification tasks. However, deep net…
Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
Younan Zhu, Linwei Tao, Minjing Dong +1
Large Vision-Language Models (LVLMs) exhibit impressive multimodal reasoning capabilities but remain highly susceptible to object hallucination, where models generate responses tha…
Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation
Yuheng Shi, Minjing Dong, Chang Xu
While Contrastive Language-Image Pre-training (CLIP) has advanced open-vocabulary predictions, its performance on semantic segmentation remains suboptimal. This shortfall primarily…
Feature Clipping for Uncertainty Calibration
Linwei Tao, Minjing Dong, Chang Xu
Deep neural networks (DNNs) have achieved significant success across various tasks, but ensuring reliable uncertainty estimates, known as model calibration, is crucial for their sa…
Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors
Linwei Tao, Haolan Guo, Minjing Dong +1
Calibration is crucial in deep learning applications, especially in fields like healthcare and autonomous driving, where accurate confidence estimates are vital for decision-making…
Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models
Jinxu Lin, Linwei Tao, Minjing Dong +1
As diffusion models become increasingly popular, the misuse of copyrighted and private images has emerged as a major concern. One promising solution to mitigate this issue is ident…
VSSD: Vision Mamba with Non-Causal State Space Duality
Yuheng Shi, Minjing Dong, Mingjia Li +1
Vision transformers have significantly advanced the field of computer vision, offering robust modeling capabilities and global receptive field. However, their high computational de…
Efficient Image-to-Image Diffusion Classifier for Adversarial Robustness
Hefei Mei, Minjing Dong, Chang Xu
Diffusion models (DMs) have demonstrated great potential in the field of adversarial robustness, where DM-based defense methods can achieve superior defense capability without adve…