Numerical influence of ReLU'(0) on backpropagation

David Bertoin, Jérôme Bolte, Sébastien Gerchinovitz +1

In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training. Yet, in the real world, 32 bits default precision c…