4 papers
Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting
Kihyun Kim, Hee-Seon Kim, Wonjun Lee +1
Batch prompting is a practical inference strategy for large language models, but its safety implications remain underexplored. We show that the success of batch prompting for utili…
Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
Hee-Seon Kim, Minbeom Kim, Seokil Ham +1
Large Vision-Language Models (LVLMs) have achieved remarkable performance on multimodal tasks but remain highly vulnerable to small adversarial perturbations in input images. Exist…
Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts
Hee-Seon Kim, Minbeom Kim, Wonjun Lee +2
Optimization-based jailbreaks typically adopt the Toxic-Continuation setting in large vision-language models (LVLMs), following the standard next-token prediction objective. In thi…
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation
Seokil Ham, Hee-Seon Kim, Sangmin Woo +1
Despite the growing interest in Mamba architecture as a potential replacement for Transformer architecture, parameter-efficient fine-tuning (PEFT) approaches for Mamba remain large…