What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
arXiv:2101.06898
Abstract
Deep neural networks (DNNs) are increasingly deployed in different applications to achieve state-of-the-art performance. However, they are often applied as a black box with limited understanding of what knowledge the model has learned from the data. In this paper, we focus on image classification and propose a method to visualize and understand the class-wise knowledge (patterns) learned by DNNs under three different settings including natural, backdoor and adversarial. Different to existing visualization methods, our method searches for a single predictive pattern in the pixel space to represent the knowledge learned by the model for each class. Based on the proposed method, we show that DNNs trained on natural (clean) data learn abstract shapes along with some texture, and backdoored models learn a suspicious pattern for the backdoored class. Interestingly, the phenomenon that DNNs can learn a single predictive pattern for each class indicates that DNNs can learn a backdoor even from clean data, and the pattern itself is a backdoor trigger. In the adversarial setting, we show that adversarially trained models tend to learn more simplified shape patterns. Our method can serve as a useful tool to better understand the knowledge learned by DNNs on different datasets under different settings.
References in corpus (7)
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
- Defending Neural Backdoors via Generative Distribution Modeling
- Clean-Label Backdoor Attacks on Video Recognition Models