The Power of Depth for Feedforward Neural Networks

Ronen Eldan, Ohad Shamir

We show that there is a simple (approximately radial) function on ${\mathbb{R}}^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-laye…