The Local Elasticity of Neural Networks

Abstract

This paper presents a phenomenon in neural networks that we refer to as \textit{local elasticity}. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector $\bx'$ is \textit{not} significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector $\bx$ that is \textit{dissimilar} to $\bx'$ in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel \citep{jacot2018neural}. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with $K$-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.

arXiv commentTo appear in ICLR 2020

References in corpus (30)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding Black-box Predictions via Influence Functions
• On the Number of Linear Regions of Deep Neural Networks
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• SGD Learns the Conjugate Kernel Class of the Network
• Implicit Regularization in Deep Matrix Factorization
• Data-Dependent Stability of Stochastic Gradient Descent
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Complexity of Linear Regions in Deep Networks
• Stiffness: A New Perspective on Generalization in Neural Networks
• Adding One Neuron Can Eliminate All Bad Local Minima
• Classification regions of deep neural networks
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• How Well Generative Adversarial Networks Learn Distributions

Cited by in corpus (5)

• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Mathematical Models of Overparameterized Neural Networks
• Understanding Deep Learning via Decision Boundary
• Toward Better Generalization Bounds with Locally Elastic Stability
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples