Efficient Search for Customized Activation Functions with Gradient Descent

Lukas Strack, Mahmoud Safari, Frank Hutter

Different activation functions work best for different deep learning models. To exploit this, we leverage recent advancements in gradient-based search techniques for neural archite…