Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

Bofeng Huang, Jacques Sun, Diane Bouchacourt +2

Web data curation has been widely studied for decoder Large Language Model (LLM) pretraining. Encoders for dense-terminology domains such as medicine, by contrast, are pretrained o…