FOGO: Forgetting-aware Orthogonalization Optimizer

Toan Nguyen, Yang Liu, Trung Le +2

We argue that forgetting is not confined to continual learning but is a general optimization phenomenon: during standard training, dominant mini-batch gradients suppress rare but u…