Angular Visual Hardness
arXiv:1912.02279
Abstract
Recent convolutional neural networks (CNNs) have led to impressive performance but often suffer from poor calibration. They tend to be overconfident, with the model confidence not always reflecting the underlying true ambiguity and hardness. In this paper, we propose angular visual hardness (AVH), a score given by the normalized angular distance between the sample feature embedding and the target classifier to measure sample hardness. We validate this score with an in-depth and extensive scientific study, and observe that CNN models with the highest accuracy also have the best AVH scores. This agrees with an earlier finding that state-of-art models improve on the classification of harder examples. We observe that the training dynamics of AVH is vastly different compared to the training loss. Specifically, AVH quickly reaches a plateau for all samples even though the training loss keeps improving. This suggests the need for designing better loss functions that can target harder examples more effectively. We also find that AVH has a statistically significant correlation with human visual hardness. Finally, we demonstrate the benefit of AVH to a variety of applications such as self-training for domain adaptation and domain generalization.
References in corpus (21)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- Learning Transferable Features with Deep Adaptation Networks
- On Calibration of Modern Neural Networks
- Learning Face Representation from Scratch
- Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
- VisDA: The Visual Domain Adaptation Challenge
- A DIRT-T Approach to Unsupervised Domain Adaptation
- Generalisation in humans and deep neural networks
- Do ImageNet Classifiers Generalize to ImageNet?
- Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
- Not All Samples Are Created Equal: Deep Learning with Importance Sampling
- Iterative Machine Teaching
- Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
- Multiple Instance Detection Network with Online Instance Classifier Refinement
- Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
- Eigen-Distortions of Hierarchical Representations
- Neural Similarity Learning
- Human perception in computer vision
- Orthogonal Over-Parameterized Training