On the Difference of BERT-style and CLIP-style Text Encoders

Zhihong Chen, Guiming Hardy Chen, Shizhe Diao +2

Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models. Recently, contrast…