On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learning

Trung Pham, Chaoning Zhang, Axi Niu +2

Exponential Moving Average (EMA or momentum) is widely used in modern self-supervised learning (SSL) approaches, such as MoCo, for enhancing performance. We demonstrate that such m…