Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning
arXiv:2103.00370
Abstract
Visual search, recommendation, and contrastive similarity learning power technologies that impact billions of users worldwide. Modern model architectures can be complex and difficult to interpret, and there are several competing techniques one can use to explain a search engine's behavior. We show that the theory of fair credit assignment provides a axiomatic solution that generalizes several existing recommendation- and metric-explainability techniques in the literature. Using this formalism, we show when existing approaches violate "fairness" and derive methods that sidestep these shortcomings and naturally handle counterfactual information. More specifically, we show existing approaches implicitly approximate second-order Shapley-Taylor indices and extend CAM, GradCAM, LIME, SHAP, SBSM, and other methods to search engines. These extensions can extract pairwise correspondences between images from trained models. We also introduce a fast kernel-based method for estimating Shapley-Taylor indices that require orders of magnitude fewer function evaluations to converge. Finally, we show that these game-theoretic measures yield more consistent explanations for image similarity architectures.
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- A Simple Framework for Contrastive Learning of Visual Representations
- Learning Transferable Visual Models From Natural Language Supervision
- Axiomatic Attribution for Deep Networks
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- InterpretML: A Unified Framework for Machine Learning Interpretability
- Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
- Cross Attention Network for Few-shot Classification
- On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
- Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
- Visual Explanation for Deep Metric Learning
- Towards Visually Explaining Similarity Models