The Optimal Condition Number for ReLU Function

Yu Xia, Haoyu Zhou

ReLU is a widely used activation function in deep neural networks. This paper explores the stability properties of the ReLU map. For any weight matrix $\boldsymbol{A} \in \mathbb{R…