Introspective Attention Modulation for Safe Text-to-Image Generation
arXiv:2607.14945
The paper proposes a method that monitors and adjusts the attention mechanisms of text‑to‑image diffusion models at inference time to prevent the generation of unsafe content while keeping image quality and semantic alignment.
Abstract
State-of-the-art flow based text-to-image (T2I) models exhibit remarkable generative abilities but remain vulnerable to producing unsafe content. Prior safety efforts range from concept erasure and prompt filtering to classifier-based gating. However, simple techniques like parameter efficient adaptations of the models easily bypass such guardrails. We introduce a unique principled approach that achieves safety by regulating the model's attention dynamics through inference-time introspection, exhibiting intrinsic robustness. Our method analyzes and rebalances attention activations throughout image synthesis, steering generations away from unsafe concepts while preserving semantic alignment. This introspective control ensures safety of deployed models. Across standard and adversarial safety benchmarks, our approach achieves remarkable safety scores while maintaining or even improving alignment and perceptual quality. Our results reveal that attention-space regulation offers a considerably more promising path to safer diffusion transformer based image generation than the existing concept erasing mechanism.Our code can be accessed at https://basim-azam.github.io/iam/
Accepted at ECCV 2026. 20 pages, 7 figures. Project page: https://basim-azam.github.io/iam/