Vision Transformers Need Registers
arXiv:2309.16588
Abstract
Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.
References in corpus (15)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- DINOv2: Learning Robust Visual Features without Supervision
- BEiT: BERT Pre-Training of Image Transformers
- Segment Anything
- Object-Centric Learning with Slot Attention
- iBOT: Image BERT Pre-Training with Online Tokenizer
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
- Localizing Objects with Self-Supervised Transformers and no Labels
- ViDT: An Efficient and Effective Fully Transformer-based Object Detector
- Recurrent Memory Transformer
- Memory Transformer
- Adaptive Computation with Elastic Input Sequence
- TOAST: Transfer Learning via Attention Steering
- Emergence of Segmentation with Minimalistic White-Box Transformers