Robustness Emerges Early in Training Dynamics, but Is Not Preserved

Jiangang Yang, Wenhui Shi, Lu Hu +2

Robustness to natural corruptions remains a fundamental challenge for deep neural networks. In this paper, we identify a robustness fading phenomenon where shallow layers spontaneo…