Adversarial Token Attacks on Vision Transformers

Abstract

Vision transformers rely on a patch token based self attention mechanism, in contrast to convolutional networks. We investigate fundamental differences between these two families of models, by designing a block sparsity based adversarial token attack. We probe and analyze transformer as well as convolutional models with token attacks of varying patch sizes. We infer that transformer models are more sensitive to token attacks than convolutional models, with ResNets outperforming Transformer models by up to $\sim 30\%$ in robust accuracy for single token attacks.

References in corpus (11)

• Language Models are Few-Shot Learners
• Wide Residual Networks
• MLP-Mixer: An all-MLP Architecture for Vision
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Spatially Transformed Adversarial Examples
• Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors
• CvT: Introducing Convolutions to Vision Transformers
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Defense Against the Dark Arts: An overview of adversarial example security research and future research directions

Cited by in corpus (1)

• DBIA: Data-free Backdoor Injection Attack against Transformer Networks