Bounding and Counting Linear Regions of Deep Neural Networks

Abstract

We investigate the complexity of deep neural networks (DNN) that represent piecewise linear (PWL) functions. In particular, we study the number of linear regions, i.e. pieces, that a PWL function represented by a DNN can attain, both theoretically and empirically. We present (i) tighter upper and lower bounds for the maximum number of linear regions on rectifier networks, which are exact for inputs of dimension one; (ii) a first upper bound for multi-layer maxout networks; and (iii) a first method to perform exact enumeration or counting of the number of regions by modeling the DNN with a mixed-integer linear formulation. These bounds come from leveraging the dimension of the space defining each linear region. The results also indicate that a deep rectifier network can only have more linear regions than every shallow counterpart with same number of neurons if that number exceeds the dimension of the input.

ICML 2018

References in corpus (2)

• On the Number of Linear Regions of Deep Neural Networks
• On the number of response regions of deep feed forward networks with piece-wise linear activations

Cited by in corpus (19)

• Efficient representation and approximation of model predictive control laws via deep learning
• Understanding Deep Neural Networks with Rectified Linear Units
• Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility Study
• Continual Learning in Low-rank Orthogonal Subspaces
• Is Deeper Better only when Shallow is Good?
• Reachability Analysis for Feed-Forward Neural Networks using Face Lattices
• Learning Strict Identity Mappings in Deep Residual Networks
• Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation
• Function approximation by deep networks
• Learning Boolean Circuits with Neural Networks
• Bounding The Number of Linear Regions in Local Area for Neural Networks with ReLU Activations
• Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks
• ReLUSyn: Synthesizing Stealthy Attacks for Deep Neural Network Based Cyber-Physical Systems
• In Proximity of ReLU DNN, PWA Function, and Explicit MPC
• Deep Networks as Logical Circuits: Generalization and Interpretation
• How Could Polyhedral Theory Harness Deep Learning?
• ReLU activated Multi-Layer Neural Networks trained with Mixed Integer Linear Programs
• How Analysis Can Teach Us the Optimal Way to Design Neural Operators
• A simple geometric proof for the benefit of depth in ReLU networks