Feature selection of neural networks is skewed towards the less abstract cue
arXiv:1908.03000
Abstract
Artificial neural networks (ANNs) have become an important tool for image classification with many applications in research and industry. However, it remains largely unknown how relevant image features are selected and how data properties affect this process. In particular, we are interested whether the abstraction level of image cues correlating with class membership influences feature selection. We perform experiments with binary images that contain a combination of cues, representing two different levels of abstractions: one is a pattern drawn from a random distribution where class membership correlates with the statistics of the pattern, the other a combination of symbol-like entities, where the symbolic code correlates with class membership. When the network is trained with data in which both cues are equally significant, we observe that the cues at the lower abstraction level, i.e., the pattern, is learned, while the symbolic information is largely ignored, even in networks with many layers. Symbol-like entities are only learned if the importance of low-level cues is reduced compared to the high-level ones. These findings raise important questions about the relevance of features that are learned by deep ANNs and how learning could be shifted towards symbolic features.
arXiv comment8 pages, 5 figures, 8 tables
References in corpus (4)
- On the Number of Linear Regions of Deep Neural Networks
- Opening the Black Box of Deep Neural Networks via Information
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- On the number of response regions of deep feed forward networks with piece-wise linear activations