Inseq: An Interpretability Toolkit for Sequence Generation Models
arXiv:2302.13942 · doi:10.18653/v1/2023.acl-demo.40
Abstract
Past work in natural language processing interpretability focused mainly on popular classification tasks while largely overlooking generation settings, partly due to a lack of dedicated tools. In this work, we introduce Inseq, a Python library to democratize access to interpretability analyses of sequence generation models. Inseq enables intuitive and optimized extraction of models' internal information and feature importance scores for popular decoder-only and encoder-decoder Transformers architectures. We showcase its potential by adopting it to highlight gender biases in machine translation models and locate factual knowledge inside GPT-2. Thanks to its extensible interface supporting cutting-edge techniques such as contrastive feature attribution, Inseq can drive future advances in explainable natural language generation, centralizing good practices and enabling fair and reproducible model evaluations.
arXiv commentACL 2023 Demo Track. Library: https://github.com/inseq-team/inseq, Docs: https://inseq.readthedocs.io, v0.4
References in corpus (18)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Semantics derived automatically from language corpora contain human-like biases
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Learning Important Features Through Propagating Activation Differences
- Scaling Laws for Neural Language Models
- Scaling Instruction-Finetuned Language Models
- Captum: A unified and generic model interpretability library for PyTorch
- Pathologies of Neural Models Make Interpretations Difficult
- Post-hoc Interpretability for Neural NLP: A Survey
- Locating and Editing Factual Associations in GPT
- Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild
- ferret: a Framework for Benchmarking Explainers on Transformers
- How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points
- Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
- Trustworthy Social Bias Measurement
- Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translation
- Reducing Hallucinations in Neural Machine Translation with Feature Attribution