A simple geometric proof for the benefit of depth in ReLU networks
arXiv:2101.07126
Abstract
We present a simple proof for the benefit of depth in multi-layer feedforward network with rectified activation ("depth separation"). Specifically we present a sequence of classification problems indexed by such that (a) for any fixed depth rectified network there exist an above which classifying problem correctly requires exponential number of parameters (in ); and (b) for any problem in the sequence, we present a concrete neural network with linear depth (in ) and small constant width () that classifies the problem with zero error. The constructive proof is based on geometric arguments and a space folding construction. While stronger bounds and results exist, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.
arXiv comment9 pages, 5 figures
References in corpus (6)
- Representation Benefits of Deep Feedforward Networks
- Understanding Deep Neural Networks with Rectified Linear Units
- On the number of response regions of deep feed forward networks with piece-wise linear activations
- Benefits of depth in neural networks
- Bounding and Counting Linear Regions of Deep Neural Networks
- Is Deeper Better only when Shallow is Good?