Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Elisa Oostwal, Michiel Straat, Michael Biehl

We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is…