Don't Lose Focus: Activation Steering via Key-Orthogonal Projections

Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik

Activation steering controls LLM behaviour towards target behaviour by intervening in internal representations, yet it often degrades reasoning and retrieval performance. We argue…