Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks

Abstract

Recent articles indicate that deep neural networks are efficient models for various learning problems. However they are often highly sensitive to various changes that cannot be detected by an independent observer. As our understanding of deep neural networks with traditional generalization bounds still remains incomplete, there are several measures which capture the behaviour of the model in case of small changes at a specific state. In this paper we consider adversarial stability in the tangent space and suggest tangent sensitivity in order to characterize stability. We focus on a particular kind of stability with respect to changes in parameters that are induced by individual examples without known labels. We derive several easily computable bounds and empirical measures for feed-forward fully connected ReLU (Rectified Linear Unit) networks and connect tangent sensitivity to the distribution of the activation regions in the input space realized by the network. Our experiments suggest that even simple bounds and measures are associated with the empirical generalization gap.

arXiv comment14 pages, 4 figures, 2 tables

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Improving Generalization Performance by Switching from Adam to SGD
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Spectrally-normalized margin bounds for neural networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Moments and Absolute Moments of the Normal Distribution
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• The power of deeper networks for expressing natural functions
• Uniform convergence may be unable to explain generalization in deep learning
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Stronger generalization bounds for deep nets via a compression approach
• Algorithmic stability and hypothesis complexity