SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

Gaurav Sarkar, Jay Gala, Subarna Tripathi

The design of activation functions remains a pivotal component in optimizing deep neural networks. While prevailing choices like Swish and GELU demonstrate considerable efficacy, t…