A simple geometric proof for the benefit of depth in ReLU networks

Asaf Amrami, Yoav Goldberg

We present a simple proof for the benefit of depth in multi-layer feedforward network with rectified activation ("depth separation"). Specifically we present a sequence of classifi…