Investigating the Compositional Structure Of Deep Neural Networks

Abstract

The current understanding of deep neural networks can only partially explain how input structure, network parameters and optimization algorithms jointly contribute to achieve the strong generalization power that is typically observed in many real-world applications. In order to improve the comprehension and interpretability of deep neural networks, we here introduce a novel theoretical framework based on the compositional structure of piecewise linear activation functions. By defining a direct acyclic graph representing the composition of activation patterns through the network layers, it is possible to characterize the instances of the input data with respect to both the predicted label and the specific (linear) transformation used to perform predictions. Preliminary tests on the MNIST dataset show that our method can group input instances with regard to their similarity in the internal representation of the neural network, providing an intuitive measure of input complexity.

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Understanding deep learning requires rethinking generalization
• On the Number of Linear Regions of Deep Neural Networks
• A Closer Look at Memorization in Deep Networks
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• On the Expressive Power of Deep Neural Networks
• Complexity of Linear Regions in Deep Networks
• Learning What Data to Learn
• Empirical Studies on the Properties of Linear Regions in Deep Neural Networks