Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions

Abstract

In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. This implies that a sufficiently wide hidden layer is necessary to guarantee that the network can produce disconnected decision regions. We discuss the implications of this result for the construction of neural networks, in particular the relation to the problem of adversarial manipulation of classifiers.

arXiv commentAccepted at ICML 2018. Added discussion for non-pyramidal networks and ReLU activation function

References in corpus (12)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• On the Number of Linear Regions of Deep Neural Networks
• Sum-Product Networks: A New Deep Architecture
• Adversarial Machine Learning at Scale
• The loss surface of deep and wide neural networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Representation Benefits of Deep Feedforward Networks
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Optimization Landscape and Expressivity of Deep CNNs
• Classification regions of deep neural networks
• A Tropical Approach to Neural Networks with Piecewise Linear Activations
• Deep vs. shallow networks : An approximation theory perspective

Cited by in corpus (5)

• Two-hidden-layer Feedforward Neural Networks are Universal Approximators: A Constructive Approach
• A Differential Topological View of Challenges in Learning with Feedforward Neural Networks
• A Topological Framework for Deep Learning
• Topological Deep Learning: Classification Neural Networks
• Deep Neural Networks Are Congestion Games: From Loss Landscape to Wardrop Equilibrium and Beyond