Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
arXiv:2001.01072
Abstract
A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.
arXiv commentInt'l. Conf. on Learning Representations (ICLR), Addis Ababa, Ethiopia, April 2020
References in corpus (8)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- On the Number of Linear Regions of Deep Neural Networks
- Exponential expressivity in deep neural networks through transient chaos
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Dropout Training as Adaptive Regularization
- Adversarial Robustness through Local Linearization
- The Shattered Gradients Problem: If resnets are the answer, then what is the question?
- Complexity of Linear Regions in Deep Networks
Cited by in corpus (9)
- Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
- Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums
- The Power of Typed Affine Decision Structures: A Case Study
- Investigating the Compositional Structure Of Deep Neural Networks
- Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
- Neural networks behave as hash encoders: An empirical study
- How to Explain Neural Networks: an Approximation Perspective
- On the Expected Complexity of Maxout Networks
- Gradient representations in ReLU networks as similarity functions