Causal Localization of the Refusal Direction in Audio Language Models

Leonardo Haw-Yang Foo, Hung-yi Lee

A large audio language model (LALM) attaches a speech front end to a text language model (LM) that is already safety-aligned. When such a model refuses a harmful spoken request, is…