6 papers
Per-parameter Task Arithmetic for Unlearning in Large Language Models
Chengyi Cai, Zesheng Ye, Jiangchao Yao +5
In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the paramet…
Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning
Chengyi Cai, Zesheng Ye, Peike Li +3
Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods larg…
Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
Zesheng Ye, Chengyi Cai, Ruijiang Dong +4
As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Des…
Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Chengyi Cai, Zesheng Ye, Lei Feng +2
Model reprogramming adapts pretrained models to downstream tasks by modifying only the input and output spaces. Visual reprogramming (VR) is one instance for vision tasks that adds…
Attribute-based Visual Reprogramming for Vision-Language Models
Chengyi Cai, Zesheng Ye, Lei Feng +2
Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language m…
Bayesian-guided Label Mapping for Visual Reprogramming
Chengyi Cai, Zesheng Ye, Lei Feng +2
Visual reprogramming (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e.…