On the Stepwise Nature of Self-Supervised Learning

James B. Simon, Maksis Knutins, Liu Ziyin +3

We present a simple picture of the training process of joint embedding self-supervised learning methods. We find that these methods learn their high-dimensional embeddings one dime…