Human Attention in Fine-grained Classification

Abstract

The way humans attend to, process and classify a given image has the potential to vastly benefit the performance of deep learning models. Exploiting where humans are focusing can rectify models when they are deviating from essential features for correct decisions. To validate that human attention contains valuable information for decision-making processes such as fine-grained classification, we compare human attention and model explanations in discovering important features. Towards this goal, we collect human gaze data for the fine-grained classification dataset CUB and build a dataset named CUB-GHA (Gaze-based Human Attention). Furthermore, we propose the Gaze Augmentation Training (GAT) and Knowledge Fusion Network (KFN) to integrate human gaze knowledge into classification models. We implement our proposals in CUB-GHA and the recently released medical dataset CXR-Eye of chest X-ray images, which includes gaze data collected from a radiologist. Our result reveals that integrating human attention knowledge benefits classification effectively, e.g. improving the baseline by 4.38% on CXR. Hence, our work provides not only valuable insights into understanding human attention in fine-grained classification, but also contributes to future research in integrating human gaze with computer vision tasks. CUB-GHA and code are available at https://github.com/yaorong0921/CUB-GHA.

19 pages, 9 figures

References in corpus (12)

• An overview of gradient descent optimization algorithms
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Recurrent Models of Visual Attention
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• A Benchmark for Interpretability Methods in Deep Neural Networks
• DeepGaze II: Reading fixations from deep features trained on object recognition
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly
• Fully Convolutional Attention Networks for Fine-Grained Recognition
• Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition
• Dynamic Computational Time for Visual Attention

Cited by in corpus (3)

• Where and What: Driver Attention-based Object Detection
• Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAI
• Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task