Adaptive wavelet distillation from neural networks through interpretations
arXiv:2107.09145
Abstract
Recent deep-learning models have achieved impressive prediction performance, but often sacrifice interpretability and computational efficiency. Interpretability is crucial in many disciplines, such as science and medicine, where models must be carefully vetted or where interpretation is the goal itself. Moreover, interpretable models are concise and often yield computational efficiency. Here, we propose adaptive wavelet distillation (AWD), a method which aims to distill information from a trained neural network into a wavelet transform. Specifically, AWD penalizes feature attributions of a neural network in the wavelet domain to learn an effective multi-resolution wavelet transform. The resulting model is highly predictive, concise, computationally efficient, and has properties (such as a multi-scale structure) which make it easy to interpret. In close collaboration with domain experts, we showcase how AWD addresses challenges in two real-world settings: cosmological parameter inference and molecular-partner prediction. In both cases, AWD yields a scientifically interpretable and concise model which gives predictive performance better than state-of-the-art neural networks. Moreover, AWD identifies predictive features that are scientifically meaningful in the context of respective domains. All code and models are released in a full-fledged package available on Github (https://github.com/Yu-Group/adaptive-wavelets).
References in corpus (17)
- Distilling the Knowledge in a Neural Network
- Towards A Rigorous Science of Interpretable Machine Learning
- Weak Gravitational Lensing
- Interpretable machine learning: definitions, methods, and applications
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
- Sanity Checks for Saliency Maps
- Distilling a Neural Network Into a Soft Decision Tree
- Cosmological constraints with deep learning from KiDS-450 weak lensing maps
- Weak lensing cosmology with convolutional neural networks on noisy data
- Constraining neutrino mass with tomographic weak lensing peak counts
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
- Weak Lensing Peak Finding: Estimators, Filters, and Biases
- Multiscale Adaptive Representation of Signals: I. The Basic Framework
- Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation
- Transformation Importance with Applications to Cosmology
- Learning Sparse Wavelet Representations
- Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup