Emergence of Human-Like Attention in Self-Supervised Vision Transformers: an eye-tracking study

Abstract

Many models of visual attention have been proposed so far. Traditional bottom-up models, like saliency models, fail to replicate human gaze patterns, and deep gaze prediction models lack biological plausibility due to their reliance on supervised learning. Vision Transformers (ViTs), with their self-attention mechanisms, offer a new approach but often produce dispersed attention patterns if trained with supervised learning. This study explores whether self-supervised DINO (self-DIstillation with NO labels) training enables ViTs to develop attention mechanisms resembling human visual attention. Using video stimuli to capture human gaze dynamics, we found that DINO-trained ViTs closely mimic human attention patterns, while those trained with supervised learning deviate significantly. An analysis of self-attention heads revealed three distinct clusters: one focusing on foreground objects, one on entire objects, and one on the background. DINO-trained ViTs offer insight into how human overt attention and figure-ground separation develop in visual perception.

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning deep representations by mutual information estimation and maximization
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• Track Anything: Segment Anything Meets Videos
• Large Associative Memory Problem in Neurobiology and Machine Learning
• Object Based Attention Through Internal Gating