Separating Representation from Reconstruction Enables Scalable Text Encoders

Megi Dervishi, Mathurin Videau, Yann LeCun

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the r…