What do Deep Networks Like to Read?
arXiv:1909.04547
Abstract
Recent research towards understanding neural networks probes models in a top-down manner, but is only able to identify model tendencies that are known a priori. We propose Susceptibility Identification through Fine-Tuning (SIFT), a novel abstractive method that uncovers a model's preferences without imposing any prior. By fine-tuning an autoencoder with the gradients from a fixed classifier, we are able to extract propensities that characterize different kinds of classifiers in a bottom-up manner. We further leverage the SIFT architecture to rephrase sentences in order to predict the opposing class of the ground truth label, uncovering potential artifacts encoded in the fixed classification model. We evaluate our method on three diverse tasks with four different models. We contrast the propensities of the models as well as reproduce artifacts reported in the literature.
References in corpus (7)
- Sequence to Sequence Learning with Neural Networks
- Categorical Reparameterization with Gumbel-Softmax
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
- Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts