Rescaling MLM-Head for Neural Sparse Retrieval

Youngjoon Jang, Seongtae Hong, Jonah Turner +1

Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT wi…