Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen +5

Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-p…