Hysteresis Activation Function for Efficient Inference

Moshe Kimhi, Idan Kashani, Avi Mendelson +1

The widely used ReLU is favored for its hardware efficiency, {as the implementation at inference is a one bit sign case,} yet suffers from issues such as the ``dying ReLU'' problem…