SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

Gaurav Sarkar, Jay Gala, Syed Affan Daimi +1

Prevailing activation functions such as Swish and GELU tend toward domain-specific optima, Swish was discovered via neural architecture search on vision benchmarks, while GELU domi…