SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning

Tyler Kastner, Nimrod De La Vega, Amir-massoud Farahmand

Neural networks trained on nonstationary tasks frequently lose the ability to fit new targets, a phenomenon referred to as loss of plasticity. We identify a novel source of plastic…