Why ReLU Units Sometimes Die: Analysis of Single-Unit Error Backpropagation in Neural Networks

Scott C. Douglas, Jiutian Yu

Recently, neural networks in machine learning use rectified linear units (ReLUs) in early processing layers for better performance. Training these structures sometimes results in "…