Deep ReLU Networks Have Surprisingly Few Activation Patterns

Abstract

The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In ReLU networks, the number of activation patterns is one measure of expressivity; and the maximum number of patterns grows exponentially with the depth. However, recent work has showed that the practical expressivity of deep networks - the functions they can learn rather than express - is often far from the theoretical maximum. In this paper, we show that the average number of activation patterns for ReLU networks at initialization is bounded by the total number of neurons raised to the input dimension. We show empirically that this bound, which is independent of the depth, is tight both at initialization and during training, even on memorization tasks that should maximize the number of activation patterns. Our work suggests that realizing the full expressivity of deep networks may not be possible in practice, at least with current methods.

arXiv comment18 page, 7 figures

Cited by in corpus (43)

• The Modern Mathematics of Deep Learning
• Review: Deep Learning in Electron Microscopy
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Neural Architecture Search without Training
• Reliably-stabilizing piecewise-affine neural network controllers
• Exactly Computing the Local Lipschitz Constant of ReLU Networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
• Reverse-Engineering Deep ReLU Networks
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Consistent Counterfactuals for Deep Models
• Deep ReLU Networks Preserve Expected Length
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Perfectly Parallel Fairness Certification of Neural Networks
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• Constrained Feedforward Neural Network Training via Reachability Analysis
• Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums
• An Embedding of ReLU Networks and an Analysis of their Identifiability
• Scaling Up Exact Neural Network Compression by ReLU Stability
• The Power of Typed Affine Decision Structures: A Case Study
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
• Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verification
• Hyperplane Arrangements of Trained ConvNets Are Biased
• Initializing ReLU networks in an expressive subspace of weights
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Switched linear projections for neural network interpretability
• Depth separation beyond radial functions
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Investigating the Compositional Structure Of Deep Neural Networks
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Detecting Out-of-distribution Objects Using Neuron Activation Patterns
• DISCO Verification: Division of Input Space into COnvex polytopes for neural network verification
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Neural networks behave as hash encoders: An empirical study
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Bounding The Number of Linear Regions in Local Area for Neural Networks with ReLU Activations
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations
• On the Expected Complexity of Maxout Networks
• Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem
• How to Explain Neural Networks: an Approximation Perspective
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks