8 papers
Disrupting Hierarchical Reasoning: Adversarial Protection for Geographic Privacy in Multimodal Reasoning Models
Jiaming Zhang, Che Wang, Yang Cao +2
Multi-modal large reasoning models (MLRMs) pose significant privacy risks by inferring precise geographic locations from personal images through hierarchical chain-of-thought reaso…
ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction
Che Wang, Fuyao Zhang, Jiaming Zhang +6
Large Language Model (LLM) agents are susceptible to Indirect Prompt Injection (IPI) attacks, where malicious instructions in retrieved content hijack the agent's execution. Existi…
VENOMREC: Cross-Modal Interactive Poisoning for Targeted Promotion in Multimodal LLM Recommender Systems
Guowei Guan, Yurong Hao, Jiaming Zhang +6
Multimodal large language models (MLLMs) are pushing recommender systems (RecSys) toward content-grounded retrieval and ranking via cross-modal fusion. We find that while cross-mod…
Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning
Zhihang Yuan, Chengyu Yue, Long Huang +2
Instruction tuning is a standard paradigm for adapting large language models (LLMs), but modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning c…
DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents
Fuyao Zhang, Jiaming Zhang, Che Wang +6
The reliance of mobile GUI agents on Multimodal Large Language Models (MLLMs) introduces a severe privacy vulnerability: screenshots containing Personally Identifiable Information…
Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models
Fuyao Zhang, Xinyu Yan, Tiantong Wu +7
Large Language Models (LLMs) increasingly leverage Federated Learning (FL) to utilize private, task-specific datasets for fine-tuning while preserving data privacy. However, while…