dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

Alois Bachmann

This paper proposes $\mathrm{dynActivation}$, a per-layer trainable activation defined as $f_i(x)=\mathrm{BaseAct}(x)({\alpha }_i-{\beta }_i)+{\beta }_{i}x$, where ${\alpha }_i$ and ${\beta }_i$ are lightweight l…