Introspective Learning by Distilling Knowledge from Online Self-explanation

Abstract

In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to improve the learning process has been less explored. As the privileged information, the explanations of a model can be used to guide the learning process of the model itself. In the community, another intensively investigated privileged information used to guide the training of a model is the knowledge from a powerful teacher model. The goal of this work is to leverage the self-explanation to improve the learning process by borrowing ideas from knowledge distillation. We start by investigating the effective components of the knowledge transferred from the teacher network to the student network. Our investigation reveals that both the responses in non-ground-truth classes and class-similarity information in teacher's outputs contribute to the success of the knowledge distillation. Motivated by the conclusion, we propose an implementation of introspective learning by distilling knowledge from online self-explanations. The models trained with the introspective learning procedure outperform the ones trained with the standard learning procedure, as well as the ones trained with different regularization methods. When compared to the models learned from peer networks or teacher networks, our models also show competitive performance and requires neither peers nor teachers.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• When Does Label Smoothing Help?
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Moonshine: Distilling with Cheap Convolutions
• Efficient Per-Example Gradient Computations
• Saliency Methods for Explaining Adversarial Attacks
• Search for Better Students to Learn Distilled Knowledge
• Semantics for Global and Local Interpretation of Deep Neural Networks
• Neural Network Memorization Dissection
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications

Cited by in corpus (1)

• Quantifying Predictive Uncertainty in Medical Image Analysis with Deep Kernel Learning