Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

Kexin Li, Wenjun Qiu, Joshua Abraham +2

Rectified linear unit (ReLU) networks can suffer from dying neurons, where units with persistently negative pre-activations produce zero outputs, blocking gradients through their a…