Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

João Coelho, Bruno Martins, João Magalhães +2

This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We b…