A Simple Cache Model for Image Recognition

Abstract

Training large-scale image recognition models is computationally expensive. This raises the question of whether there might be simple ways to improve the test performance of an already trained model without having to re-train or fine-tune it with new data. Here, we show that, surprisingly, this is indeed possible. The key observation we make is that the layers of a deep network close to the output layer contain independent, easily extractable class-relevant information that is not contained in the output layer itself. We propose to extract this extra class-relevant information using a simple key-value cache memory to improve the classification performance of the model at test time. Our cache memory is directly inspired by a similar cache model previously proposed for language modeling (Grave et al., 2017). This cache component does not require any training or fine-tuning; it can be applied to any pre-trained model and, by properly setting only two hyper-parameters, leads to significant improvements in its classification performance. Improvements are observed across several architectures and datasets. In the cache component, using features extracted from layers close to the output (but not from the output layer itself) as keys leads to the largest improvements. Concatenating features from multiple layers to form keys can further improve performance over using single-layer features as keys. The cache component also has a regularizing effect, a simple consequence of which is that it substantially increases the robustness of models against adversarial attacks.

arXiv commentPublished as a conference paper at NIPS 2018

References in corpus (10)

• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Densely Connected Convolutional Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Learning to Remember Rare Events
• DeepFool: a simple and accurate method to fool deep neural networks
• Analyzing the Robustness of Nearest Neighbors to Adversarial Examples
• Unbounded cache model for online language modeling with open vocabulary
• Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples

Cited by in corpus (4)

• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• Memory Augmented Neural Model for Incremental Session-based Recommendation
• Improving the robustness of ImageNet classifiers using elements of human visual cognition
• Query the model: precomputations for efficient inference with Bayesian Networks