Smooth activations and reproducibility in deep networks

Abstract

Deep networks are gradually penetrating almost every domain in our lives due to their amazing success. However, with substantive performance accuracy improvements comes the price of \emph{irreproducibility}. Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems. The popular Rectified Linear Unit (ReLU) activation has been key to recent success of deep networks. We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks. We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs. We propose a new family of activations; Smooth ReLU (\emph{SmeLU}), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap. SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility. We generalize SmeLU to give even more flexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctified Smooth Continuous Unit (RESCU) activations. Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular.

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Gaussian Error Linear Units (GELUs)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Activation Functions: Comparison of trends in Practice and Research for Deep Learning
• On the Number of Linear Regions of Deep Neural Networks
• Large scale distributed neural network training through online distillation
• Continuously Differentiable Exponential Linear Units
• Deep Learning with S-shaped Rectified Linear Activation Units
• Critical Learning Periods in Deep Neural Networks
• Evolving Normalization-Activation Layers
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• Anti-Distillation: Improving reproducibility of deep networks
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning

Cited by in corpus (3)

• Smooth Adversarial Training
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Synthesizing Irreproducibility in Deep Networks