4 papers · 1 filter
The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties
Haoyu Zhang, Yi Feng, Shibo Zheng +8
We show that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attac…
Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks
Haoyu Zhang, Xiangchen Guan, Shibo Zheng +2
We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrela…
Blind, Not Weak: A Best-of-Suite Safety-Utility Frontier for Recover-and-Reguard Defenses Against Encoded VLM Jailbreaks
Haoyu Zhang, Zhuoxi Wang, Shibo Zheng +9
Safety classifiers ("guards") are the dominant black-box defense for vision-language models, yet a guard judges an input's surface form, not its meaning: a harmful request re-encod…
Depth, Not Breadth: Best-of-N Jailbreaking Beyond Surface Noise
Haoyu Zhang, Shibo Zheng, Hanwen Liu +9
Best-of-N jailbreaking spends a query budget on surface variation, scrambling and recasing a request until one draw lands. We ask what the same search buys when it wraps a structur…