#language model pretraining

REALM: Retrieval-Augmented Language Model Pre-Training

Kelvin Guu, Kenton Lee, Zora Tung +2

The paper introduces REALM, a language model that retrieves relevant documents from a large corpus during pre‑training and inference, enabling it to use external knowledge for task…