Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)

G. Zhang, H. Li

Recently, self-normalizing neural networks (SNNs) have been proposed with the intention to avoid batch or weight normalization. The key step in SNNs is to properly scale the expone…