Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

Shivaprasad B. Gowda, Sparsh Mittal

Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation…