Interpreting and Boosting Dropout from a Game-Theoretic View
arXiv:2009.11729
Abstract
This paper aims to understand and improve the utility of the dropout operation from the perspective of game-theoretic interactions. We prove that dropout can suppress the strength of interactions between input variables of deep neural networks (DNNs). The theoretic proof is also verified by various experiments. Furthermore, we find that such interactions were strongly related to the over-fitting problem in deep learning. Thus, the utility of dropout can be regarded as decreasing interactions to alleviate the significance of over-fitting. Based on this understanding, we propose an interaction loss to further improve the utility of dropout. Experimental results have shown that the interaction loss can effectively improve the utility of dropout and boost the performance of DNNs.
References in corpus (14)
- Improving neural networks by preventing co-adaptation of feature detectors
- Axiomatic Attribution for Deep Networks
- Consistent Individualized Feature Attribution for Tree Ensembles
- Dropout Training as Adaptive Regularization
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Hierarchical interpretations for neural network predictions
- Dropout as data augmentation
- Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
- Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
- Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
- Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection
- Technical Note: Game-Theoretic Interactions of Different Orders
- Building Interpretable Interaction Trees for Deep NLP Models
- Interpreting Multivariate Shapley Interactions in DNNs