Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders

Abstract

With the recent addition of Retrieval-Augmented Generation (RAG), the scope and importance of Information Retrieval (IR) has expanded. As a result, the importance of a deeper understanding of IR models also increases. However, interpretability in IR remains under-explored, especially when it comes to the models' inner mechanisms. In this paper, we explore the possibility of adapting Integrated Gradient-based methods in an IR context to identify the role of individual neurons within the model. In particular, we provide new insights into the role of what we call "relevance" neurons, as well as how they deal with unseen data. Finally, we carry out an in-depth pruning study to validate our findings.

Accepted at ICTIR 2024

References in corpus (8)

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• Locating and Editing Factual Associations in GPT
• Declarative Experimentation in Information Retrieval using PyTerrier
• Simplified Data Wrangling with ir_datasets
• How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations
• CausaLM: Causal Model Explanation Through Counterfactual Language Models