Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval

Ohad Rubin, Jonathan Berant

Retrieval-augmented language models (LMs) have received much attention recently. However, typically the retriever is not trained jointly as a native component of the LM, but added…