Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval

Guangyuan Ma, Xing Wu, Zijia Lin +1

Masked auto-encoder pre-training has emerged as a prevalent technique for initializing and enhancing dense retrieval systems. It generally utilizes additional Transformer decoder b…