On Masked Pre-training and the Marginal Likelihood

Pablo Moreno-Muñoz, Pol G. Recasens, Søren Hauberg

Masked pre-training removes random input dimensions and learns a model that can predict the missing values. Empirical results indicate that this intuitive form of self-supervised l…