Testing Stationarity Concepts for ReLU Networks: Hardness, Regularity, and Robust Algorithms

Lai Tian, Anthony Man-Cho So

We study the computational problem of the stationarity test for the empirical loss of neural networks with ReLU activation functions. Our contributions are: Hardness: We show that…