6 papers · 1 filter
Reprogramming Vision-Language Models via Structured Prompt Reparameterization
Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari +3
Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In vision-language models, existi…
Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models
Yunbei Zhang, Shuaicheng Niu, Chengyi Cai +2
Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limite…
Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation
Yunbei Zhang, Chengyi Cai, Feng Liu +1
Adapting closed-box service models (i.e., APIs) for target tasks typically relies on reprogramming via Zeroth-Order Optimization (ZOO). However, this standard strategy is known for…
Per-parameter Task Arithmetic for Unlearning in Large Language Models
Chengyi Cai, Zesheng Ye, Jiangchao Yao +5
In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the paramet…
Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning
Chengyi Cai, Zesheng Ye, Peike Li +3
Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods larg…
Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models
Yuhao Sun, Chengyi Cai, Jiacheng Zhang +3
Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-langua…