Distilling Knowledge from Deep Networks with Applications to Healthcare Domain

Abstract

Exponential growth in Electronic Healthcare Records (EHR) has resulted in new opportunities and urgent needs for discovery of meaningful data-driven representations and patterns of diseases in Computational Phenotyping research. Deep Learning models have shown superior performance for robust prediction in computational phenotyping tasks, but suffer from the issue of model interpretability which is crucial for clinicians involved in decision-making. In this paper, we introduce a novel knowledge-distillation approach called Interpretable Mimic Learning, to learn interpretable phenotype features for making robust prediction while mimicking the performance of deep learning models. Our framework uses Gradient Boosting Trees to learn interpretable features from deep learning models such as Stacked Denoising Autoencoder and Long Short-Term Memory. Exhaustive experiments on a real-world clinical time-series dataset show that our method obtains similar or better performance than the deep learning models, and it provides interpretable phenotypes for clinical decision making.

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Theano: new features and speed improvements
• Learning to Diagnose with LSTM Recurrent Neural Networks

Cited by in corpus (30)

• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
• RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
• Automated data processing and feature engineering for deep learning and big data applications: a survey
• Reinforcement Learning in Healthcare: A Survey
• Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets
• Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
• Dropout Feature Ranking for Deep Learning Models
• Detecting Statistical Interactions from Neural Network Weights
• Model Distillation for Revenue Optimization: Interpretable Personalized Pricing
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• SLEEPER: interpretable Sleep staging via Prototypes from Expert Rules
• Machine Learning and Visualization in Clinical Decision Support: Current State and Future Directions
• Medical Diagnosis From Laboratory Tests by Combining Generative and Discriminative Learning
• Bioinformatics and Medicine in the Era of Deep Learning
• Regional Tree Regularization for Interpretability in Black Box Models
• Adversarial Attacks and Defenses: An Interpretation Perspective
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation
• TRACER: A Framework for Facilitating Accurate and Interpretable Analytics for High Stakes Applications
• Fuzzy Logic Interpretation of Quadratic Networks
• Neural-to-Tree Policy Distillation with Policy Improvement Criterion
• Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
• Absolute Value Constraint: The Reason for Invalid Performance Evaluation Results of Neural Network Models for Stock Price Prediction
• Explainability via Responsibility
• A Generalized Meta-loss function for regression and classification using privileged information