Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models

Saketh Bachu, Erfan Shayegani, Rohit Lal +6

Vision-language models (VLMs) have improved significantly in their capabilities, but their complex architecture makes their safety alignment challenging. In this paper, we reveal a…