Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-lan…