Activation Function Design Sustains Plasticity in Continual Learning

Lute Lillo, Nick Cheney

In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and op…